3D Molecule Generation via Diffusion Model with a Self-Attention-Based EGNN

Abstract

The discovery of new drugs is of great significance to human health. The diffusion model has emerged as a powerful tool for generating 3D molecular structures, achieving significant success in various applications. However, existing models lack direct interatomic confinement features and are difficult to capture long-range dependencies in macromolecules. These issues lead to the generation of molecular structures that are either inaccurate or lack rationality. To address the above issues, this paper proposes a novel 3D Molecule Generation via Diffusion Model with a self-attention-based E­(n) equivariant graph neural network (EGNN) named MGDM-Sa, which incorporates a dual equivariant score neural network DualESNet, to capture the global graph features and the local atomic environment. Especially, in DualESNet, we design a novel equivariant encoder eEncoder, which integrates EGNN and self-attention. During message propagation, EGNN ensures the equivariation of geometric features, and self-attention layer captures the long-range dependencies in the molecular graph. Thus, this design enhances the representation capability of MGDM-Sa. The experimental results demonstrate that MGDM-Sa is capable of generating valid, unique, novel, stable, and diverse drug-like molecules, thereby underscoring its potential to accelerate the drug discovery process.

1. Introduction

Innovative drugs represent a critical approach to addressing human diseases. However, the cost associated with designing new drugs is prohibitively high. On average, developing a new drug from discovery to preclinical testing, clinical trials, approval, and marketing takes approximately 10 years and incurs an expense of US $2.6 billion, with a return on investment of less than 2%. With the rapid development of deep learning, various deep generative models have shown great potential in generating stable and diverse molecules with desired properties and have achieved great success in the de novo generation of drug-like molecules with desired properties.

In recent years, deep-learning-based molecular generation methods have mainly focused on three categories: SMILES-based methods, two-dimensional molecular graph representation methods, and three-dimensional (3D) molecular graph representation methods. SMILES is a representation method that converts molecules into ASCII strings based on predefined atomic rules. Due to the sequential nature of SMILES representation, natural language processing algorithms can be effectively applied to molecular generation tasks. Therefore, researchers have adopted deep learning models, including recurrent neural networks (RNN), long short-term memory networks (LSTM), generative adversarial networks (GAN), and generative pretraining (GPT), to generate molecules based on SMILES representation. In molecular graphs, atoms are represented as nodes and chemical bonds are represented as edges. The types of atoms and chemical bonds correspond to the labels of the nodes and edges, respectively. Representative methods for generating molecules based on molecular graphs using deep generative models include VGAE, JTVAE, MolGAN, and GraphNVP. However, neither molecular graphs nor SMILES take into account the spatial conformation of compound molecules.

3D molecular conformation determines the biological, chemical, and physical properties of the molecule, such as charge distribution, potential energy, molecular docking, pharmacophore search, etc. In 3D molecular generation, capturing the 3D spatial structure features is crucial. In recent years, with the rapid development of geometric deep learning in processing geometric data, deep generative models for 3D molecular conformations have gradually become a research hotspot, for instance, G-SchNet, 3DMolNet, ENF, G-SphereNet, MolGym, Uni-Mol, 3DLinker, GeoMol, ConfVAE, etc. It is evident that 3D molecular structure design leveraging deep learning has experienced rapid development and achieved remarkable success.

By combining deep learning with statistical physics, diffusion models can efficiently generate molecules with complex 3D structures and can be applied to novel drug design tasks. To address the challenge of maintaining equivariance in the 3D space, two strategies are primarily adopted. One is to leverage the equivariance of spatial geometric structures, with representative works including MDM and 3D-EDiffMG. The other is to utilize the E(3) equivariance of intermediate geometric elements, with representative works including GFMDiff, EDM, EEGSDE, DiffBridges, MolDiff, Hier-Diff, and GEOLDM. However, the existing diffusion models for 3D molecular generation lack direct interatomic constraint features, making it difficult to capture long-range dependencies in molecules. This results in suboptimal generated structures, thereby limiting the stability and diversity of the generated molecules.

To address the above problems, we propose a 3D Molecule Generation via Diffusion Model with a self-attention-based EGNN, named MGDM-Sa, for molecule generation. In MGDM-Sa, the diffusion process is defined as a Markov chain, wherein Gaussian noise is systematically introduced to the data across a sequence of T steps. To process the global graph-level features and the local atomic environment, the reverse process is to estimate the added noise by a dual equivariant score neural network, named DualESNet. Specifically, DualESNet is composed of three components: edge construction, two equivariant encoders named eEncoder, and two scoring blocks. The edge constructor constructs a long-distance encoder and a short-distance encoder to deal with different interatomic forces, respectively. eEncoder is designed as an equivariant encoder implemented by an improved EGNN combined with a self-attention decoder. eEncoder not only acquires node features and atomic coordinates through the message-passing mechanism but also overcomes the problem of remote information loss in the native EGNN. Scoring blocks aim to estimate the gradient of the log density of atomic features and map it into a score vector. The experiments are conducted on two molecular data sets: QM9 and GEOM-Drug. The experimental results demonstrate that MGDM-Sa outperforms the baseline model in terms of performance.

The main contributions of our work are as follows:

• For 3D molecular generation, we propose a novel diffusion model named MGDM-Sa, which incorporates a novel dual equivariant score neural network named DualESNet, to capture global and local information on molecular graphs and satisfy the E(3) equivariant property. It can generate effective and stable 3D structural molecules.

• We design an equivariant encoder named eEncoder, which utilizes EGNN to ensure the equivariance of geometric features and employs self-attention layer to capture long-range dependencies within molecular graphs.

• The experimental results show that MGDM-Sa can generate geometric molecules and molecules with expected properties. Moreover, its performance is superior to that of the baseline models. Additionally, we have demonstrated that the molecules generated by MGDM-Sa have the potential to be candidate drugs.

The rest of this article is organized as follows. Section provides background information and discusses related work. Section presents the diffusion model basis of our proposed model MGDM-Sa. Section presents the detailed design of our proposed model MGDM-Sa. The experimental results and the analyses of molecules generated by MGDM-Sa are included in Sections and 6, respectively. Finally, Section concludes this article with some discussions.

2. Related Work

Generative models based on 1D strings or 2D molecular graphs have been well established. However, these generative models fail to account for the spatial conformations of compound molecules. 3D molecular conformations encompass geometric information such as position as well as important physical principles such as symmetry. In comparison with 1D string and 2D molecular graph representations, 3D geometric representations substantially broaden the molecular space to be explored while simultaneously increasing the complexity of the task. The primary challenge lies in ensuring that the model exhibits equivariance under rotation and translation operations in the 3D space. Currently, research on molecular geometric representation primarily investigates isotropy through two strategies: E(3)-Invariant and E(3)-Equivariant. Next, we delve into the methodologies for generating 3D molecules under diffusion model frameworks with these two strategies.

2.1. E­(3)-Invariant

One approach is to leverage E(3)-invariant 3D information by utilizing intermediate geometric elements, such as interatomic distances and angular data. Notable works in this domain include SchNet, PhysNet, and DimNet. MDM employs SchNet for generating 3D molecular structures while enhancing the interatomic connections that are insufficiently represented in the 3D point cloud representation of molecules. MDM introduces separate equivariant encoders to capture interatomic forces of varying strengths. Additionally, MDM incorporates a distribution control variable at each diffusion-inversion step to enhance the molecular diversity. Inspired by MDM, 3D-EDiffMG introduces long-range dependence capture equivariant encoders to encode bonding and nonbonding information based on strong and weak atomic interactions. Furthermore, 3D-EDiffMG integrates a gated multilayer perceptron module with minimal attention mechanisms to explicitly model complex long-sequence-feature interactions and long-range dependencies. As a result, 3D-EDiffMG is capable of generating valid, stable, novel, unique, and diverse drug-like molecules.

Although our proposed method is grounded in the MDM framework, it outperforms existing methods by leveraging an equivariant strategy for constructing the diffusion model, thereby achieving superior performance.

2.2. E­(3)-Equivariant

The other is the equivariant strategy, whose representative works include SE(3)-Transformers, EGNN, etc. EGNN ensures the equivariance of geometric features in the process of message propagation when dealing with graph structure data. Therefore, EGNN often plays a key role in molecular generation tasks. So far, there are many excellent frameworks that combine SE(3)-Transformers or EGNN for molecular generation tasks. For instance, GFMDiff is a generative diffusion model for molecules based on SE(3)-Transformers. GFMDiff learns the 3D geometry of molecules by acquiring features from atom features, pairwise distance features, and three-way angle features and predicts atom types, valences, and coordinates. In addition, the GFMDiff model introduces GFLoss, which minimizes the difference between the valence predicted by the dualTrack transformer network (DTN) and the valence calculated from molecular geometry. DTN combined with geometrically facilitated loss achieves good molecular and atomic stability of the generated molecules.

Currently, many diffusion model methods have achieved 3D molecule generation tasks by adopting an EGNN. EDM is the first diffusion model that uses EGNN to generate 3D molecules. However, EDM is incapable of learning the valence of different atom types and the constraint relationships between neighboring atoms, which govern bond formation and distribution. Consequently, the issue of inconsistent atom types and bonds arises, resulting in molecular instability. Since then, a variety of models based on EGNN have been proposed with the aim of enhancing the performance of EDM. For instance, EEGSDE uses the stochastic differential equations framework for capturing the dependencies between two variables and generating molecules with the desired properties. DiffBridges introduced an energy function into the diffusion bridge along with molecular geometry statistics calculated from the data. However, the problem of atom types and bond inconsistencies that cause molecular instability remains. MolDiff implements a two-stage diffusion model in order to solve the problem of inconsistent atom types and bonds. HierDiff and GEOLDM avoid the atomic bond inconsistency problem by using EGNN to embed structures into smaller domains.

The aforementioned method directly employs EGNN. In light of the critical importance of capturing long-range dependencies, our proposed MGDM-Sa enhances EGNN by incorporating the Transformer encoder into the EGNN layer, which consequently leads to a further improvement in model performance.

3. Preliminary

In this section, we present the diffusion model basis of the method proposed in this paper. MGDM-Sa leverages DDPM to perturb the data through a sequence of progressively increasing Gaussian noise levels and simultaneously estimates the score function for all noisy data distributions by training a deep neural network model conditioned on the noise magnitude. The DDPM is formulated as two Markov chains: the diffusion process and the reverse process.

3.1. Diffusion Process

Let $\mathcal{M}=(A,R)$ represents 3D molecular geometry. Here, the A ∈ {0,1} ^n×f represents the atomic features matrix, including atomic types and atomic charges. $R\in {\mathbb{R}}^{n\times 3}$ represents the atomic coordinates.

Given an initial molecular geometry ${\mathcal{M}}_0\sim q({\mathcal{M}}_0)$ , we define a fixed Markov chain over discrete time steps t ∈ {1, ..., T}. The forward process gradually perturbs the molecular conformation through Gaussian transitions controlled by a cosine variance schedule (β₁, ..., β _T ), β _t ∈ (0, 1).

$$q({\mathcal{M}}_t|{\mathcal{M}}_{t-1})=\mathbb{N}({\mathcal{M}}_t;\sqrt{1-{\beta }_t}{\mathcal{M}}_{t-1},{\beta }_{t}I)$$ 1

where $\mathbb{N}$ indicates that ${\mathcal{M}}_t$ follows a normal distribution with a mean of $\sqrt{1-{\beta }_t}{\mathcal{M}}_{t-1}$ and a variance of β _t I. In the subsequent eqs and , $\mathbb{N}$ also represents a normal distribution. The joint distribution conditioned on ${\mathcal{M}}_0$ satisfies the Markov property

$$q({\mathcal{M}}_{1:T}|{\mathcal{M}}_0)=\prod _{t=1}^{T}q({\mathcal{M}}_t|{\mathcal{M}}_{t-1})$$ 2

Following ref , for all t ∈ {0, 1, ..., T}, this Gaussian transition kernel allows us to marginalize the joint distribution in eq to obtain an analytical form for $q({\mathcal{M}}_t|{\mathcal{M}}_0)$ . For α _t : = 1 – β _t and ${\mathrm{\alpha ̅}}_t:=\prod _{s=1}^t{\alpha }_s$ , at any time step, the data can be transformed by a reparameterization trick as follows:

$$q({\mathcal{M}}_t|{\mathcal{M}}_0)=\mathbb{N}({\mathcal{M}}_t;\sqrt{{\mathrm{\alpha ̅}}_t}{\mathcal{M}}_0,(1-{\mathrm{\alpha ̅}}_t)I)$$ 3

If t → ∞, then $\sqrt{{\mathrm{\alpha ̅}}_t}\to 0,(1-{\mathrm{\alpha ̅}}_t)\to 1$ . So as the step size t gradually increases, the final distribution will be closer to the standard Gaussian distribution.

3.2. Reverse Process

To generate novel molecular conformations, the reverse process progressively removes noise from an initial Gaussian prior ${\mathcal{M}}_T\sim \mathbb{N}(0,\mathbf{I})$ , through a learned Markov chain. The joint denoising distribution is factorized as

$$p_{\theta }({\mathcal{M}}_{0:T-1}|{\mathcal{M}}_T)=\prod _{t-1}^T{p}_{\theta }({\mathcal{M}}_{t-1}|{\mathcal{M}}_t)$$ 4

$p_{\theta }({\mathcal{M}}_{t-1}|{\mathcal{M}}_t)$ is reverse transition, which follows a parameterized Gaussian distribution

$$p_{\theta }({\mathcal{M}}_{t-1}|{\mathcal{M}}_t)=\mathbb{N}({\mathcal{M}}_{t-1};{\mu }_{\theta }({\mathcal{M}}_t,t),{\sigma }_t^{2}I)$$ 5

where ${\mu }_{\theta }({\mathcal{M}}_t,t)$ is the denoising function parameterized by the neural network and σ _t is the time-step dependent variance.

Following ref , we parameterize the denoising mean μ_θ using a noise prediction network $\mathit{ϵ}$ _θ:

$${\mathit{\mu }}_{\theta }({\mathcal{M}}_t,t)=\frac{1}{\sqrt{1-{\beta }_t}}({\mathcal{M}}_t-\frac{{\beta }_t}{\sqrt{1-{\mathrm{\alpha ̅}}_t}}{\mathit{ϵ}}_{\theta }({\mathcal{M}}_t,t))$$ 6

Set σ to $\frac{{\beta }_t}{\sqrt{1-{\mathrm{\alpha ̅}}_t}}$ .The − $\mathit{ϵ}$ _θ·σ score can be treated as the score, so S _θ = − $\mathit{ϵ}$ _θ·σ is set. It is the log density of the data points at each time step, pointing in the direction of the largest growth rate of the probability density function. Since the reverse process removes the noisy component added to the data during the diffusion process at each time step, it is equivalent to a traversal of the data distribution, starting from low-density regions and progressively moving toward high-density regions guided by the log gradient. Thus, we can use S _θ in all relevant formulas in the following sections. We parameterize μ_θ to

$${\mu }_{\theta }({\mathcal{M}}_t,t)=\frac{1}{\sqrt{1-{\beta }_t}}({\mathcal{M}}_t+{\beta }_t·s_{\theta }({\mathcal{M}}_t,t))$$ 7

4. Methodology

In this section, we present the detailed design of our proposed model MGDM-Sa.

4.1. Overview

Figure a illustrates our proposed model MGDM-Sa, which is a framework for the 3D molecular generation diffusion model for drug design. The framework of MGDM-Sa is implemented by denoising diffusion probabilistic models (DDPM). MGDM-Sa consists of two modules: the diffusion process and the reverse process. The diffusion process is formally defined as a Markov chain, wherein Gaussian noise is systematically introduced into the data across a sequence of T steps. The reverse process is to estimate the added noise by the dual equivariant score neural network, named DualESNet, thereby generating new molecules from the training data. DualESNet concurrently processes the global graph-level features and the local atomic environment to enhance the relationship modeling between nodes. Specially, DualESNet is composed of three components: edge construction, two equivariant encoders eEncoders, and two scoring blocks. Edge constructor, shown in Figure b, is to extract features of edges in the molecular graph and construct long and short edge representations according to the bond length distribution characteristic. eEncoder, shown in Figure c, integrates the EGNN and self-attention layer. EGNN ensures the equivariation of geometric features, while the self-attention mechanism addresses its limitation in modeling long-range dependencies. eEncoder effectively fuses global and local information to comprehensively capture complex relationships and dependencies in molecular graphs. Scoring blocks, shown in Figure d, aim to estimate the gradient of the log density of atomic features and map it to a score vector.

1.

Our proposed model comprises two main components: the diffusion process and the reverse process. (a) Framework of MGDM-Sa, which implements the diffusion model. (b) Edge encoder, which processes the edge information on molecular graphs. (c) Equivariant encoder eEncoder, whichupdates the atomic and coordinate features of the ${\mathcal{M}}_t$ . (d) Scoring block.

4.2. DualESNet

The 3D generated molecular geometry is represented in the form of a point cloud, which requires rotation and translation equivariance. Therefore, MGDM-Sa should set an equivariant Markov kernel according to this property. Köhler et al. proposed equivariant invertible functions that transform one invariant distribution into another. Xu et al. demonstrated that Markov chains initialized with invariant priors and evolving via equivariant Markov kernels can induce invariant marginal distributions. Formally, setting $\mathbb{T}$ is a rotation or translation operation. $p({\mathbf{x}}_T)=p(\mathbb{T}({\mathbf{x}}_T))$ , $p_{\theta }({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)=p_{\theta }(\mathbb{T}({\mathbf{x}}_{t-1})|\mathbb{T}({\mathbf{x}}_t))$ . Furthermore, ref proved that the sample distribution is guaranteed to be invariant to x _T , that is $p_0(\mathbf{x})=p_0(\mathbb{T}(\mathbf{x}))$ . Thus, provided that the prior and transition kernels exhibit identical invariance properties, it is possible to construct a diffusion model capable of generating molecular conformations invariant with rotation and translation.

As shown in Figure , DualESNet is composed of three components: edge construction, two eEncoders, and two scoring blocks. We will present the detailed principles of DualESNet and the workflow in the following.

4.3. Edge Constructor

The generated 3D molecular geometry is represented as a point cloud. How to model the edge of molecular conformation generation is of great significance for molecular generation. For instance, ConfGF uses Langevin dynamics sampling to ensure rotation and translation. However, ConfGF only considers the local distances and ignores the long-range intramolecular nonbond interactions, such as van der Waals forces. EDM introduced fully connected edges. However, in EDM, all interatomic interactions are treated equally, resulting in the loss of covalent bond specificity and interference with redundant information. To ensure the effective processing of subsequent equivariant kernels, we systematically construct the connecting edges between point clouds, by using the method proposed by ref

The basic idea of the edge constructor is that the threshold radius τ = 2 Å is set according to the bond length distribution characteristics. Such that a series of graph representations are generated after screening based on atomic connections inside and outside the radius of the intramolecular interaction force. These representations are then encoded through an edge encoder, shown in Figure b. The edge encoder is responsible for encoding the chemical bond features into a vector form. Specifically, the module processes two types of key information: (1) discrete key types are mapped to fixed-dimensional vectors by embedding layers; and (2) continuous bond length features are projected through linear layers. Thus, the bond type and bond distance information are integrated through the feature fusion layer to form a representation vector containing the complete characteristics of chemical bonds, which provides parsable input for subsequent graph neural networks. Mathematically, this procedure of the edge encoder is formulated as follows.

$$h_{e_{ij}}={\mathrm{M}\mathrm{L}\mathrm{P}}_e(d_{ij})\odot e_{\mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e}}$$ 8

where MLP _e is a multilayer perceptron structured as (Linear, ReLU, Linear), d _ij = ∥r _i – r _j ∥² is the Euclidean distance between atomic coordinates of node i and j, and e _type is the embedding of edge type, which is determined by the distance between atoms and chemical properties. h _{e ij} denotes the edge feature between the ith atom and the jth atom. This design not only preserves the discrete chemical characteristics of covalent bonds but also incorporates continuous spatial information on geometric constraints to realize multidimensional modeling of molecular interactions.

4.4. eEncoders

For the long edge set and the short edge set from the edge constructor, we design a dual equivalent encoder, consisting of a long-distance equivalent encoder and a short-distance equivalent encoder. Both long-distance equivariant encoders and short-distance equivariant encoders have the same structure as that of eEncoder, which is an equivariant Markov kernel. The 3D molecular structure has denaturation, such as translation and rotation. For this, we introduce EGNN to achieve the equivariant property in eEncoder. Furthermore, to enhance the representation capabilities of the MGMD-Sa model, we consider incorporating the attention mechanism into EGNN, to capture the relationship and dependency between nodes. As illustrated in Figure c, eEncoder consists of M layers of EGNN and N self-attention layers. Specifically, when training the MGMD-Sa on the QM9 data set, the structure of eEnconder is as follows:

$$\{\begin{array}{l}eEncode{r}_l:(S{a}_1,E{g}_1,E{g}_2,S{a}_2,E{g}_3,E{g}_4,S{a}_3,E{g}_5,E{g}_6)\\ eEncode{r}_s:(S{a}_1,E{g}_1,E{g}_2,S{a}_2,E{g}_3,E{g}_4,S{a}_3,E{g}_5,E{g}_6)\end{array}$$ 9

where Sa represents the self-attention layer and Eg represents the EGNN layer. Here, both eEncoder _l and eEncoder _s are composed of 3 self-attention layers and 6 EGNN layers. When training the MGMD-Sa GEOM-Drug data set, eEnconder is as follows:

$$\{\begin{array}{l}eEncode{r}_l:(S{a}_1,E{g}_1,...,E{g}_6)\\ eEncode{r}_s:(S{a}_1,E{g}_1,...,E{g}_4)\end{array}$$ 10

Here, eEncoder _l consists of 1 self-attention layer and 6 EGNN layers, while the eEncoder _s consists of 1 self-attention layer and 4 EGNN layers. eEncoder takes nodes embeddings h = {h ₀, ..., h _n–1}, coordinate embeddings r = {r ₀, ..., r _n–1}, and side information e = {h _{e ij} |i,j = 0, ..., n – 1} as input. The detailed procedures for eEncoder are as follows.

(1) EGNN layer: We take the lth EGNN layer as an example. It takes h ^(l–1), r ^(l–1) and h _e as input. In detail, the processes of this EGNN layer are as follows.

Step 1: The message m _ij from node i to node j is computed by

$$m_{ij}^l={\mathrm{M}\mathrm{L}\mathrm{P}}_e(h_i^{(l-1)},h_j^{(l-1)},\parallel r_i^{(l-1)}-r_j^{(l-1)}{\parallel }^2,h_{e_{ij}}^{(l-1)})$$ 11

where e _ij is the feature of the edge at the (l – 1)­th layer, and MLP _e is the multilayer perceptron of the edge used to compute the message m _ij .

Step 2: The weight w _ij between the solution nodes i and j is calculated as follows.

$$w_{ij}^l=\sigma ·({\varphi }_w(m_{ij}^l))$$ 12

where MLP _w is the multilayer perceptron, and σ is the Sigmoid activation function used to convert the output of MLP _w into weights.

Step 3: The message of node ith at lth layer is aggregated by

$$m_i^l=\sum _{j\in \mathcal{N}(i)}{w}_{ij}^l·m_{ij}^l$$ 13

where $\mathcal{N}(i)$ is the set of neighbor nodes of node i.

Step 4: Output the updated atomic coordinates along with the corresponding nodes. The feature representation h _i of node i can be expressed as follows:

$$h_i^l={\mathrm{M}\mathrm{L}\mathrm{P}}_h(h_i^{(l-1)},m_i^l)$$ 14

where h _i and h _i are the feature representations of node i at layer l and layer (l – 1), respectively. MLP _h is a multilayer perception that fuses the feature representation of the current layer and aggregates messages to generate the feature representation of the next layer.

For the positional updating of node i, we consider the relative position difference between node i and all its neighboring nodes. The coordinate of node i at lth layer is updated by

$$r_i^l=r_i^{(l-1)}+\sum _{j\ne i,j\in \mathcal{N}(i)}(r_i^{(l+1)}-r_j^{(l+1)})·m_{ij}^l$$ 15

where r _i and r _i are the local representations of node i at lth layer and (l – 1)­th layer, respectively. m _ij is the message between node i and its neighbor node j at layer (l + 1).

(2) Self-attention layer: To capture the relationship and dependency between nodes, we consider incorporating the attention mechanism into EGNN. The attention mechanism is an improved multihead attention mechanism, which aggregates the features of different nodes by calculating the similarity between queries, keys, and values. Mathematically, our proposed self-attention can be described by

$$h_i^l=\mathrm{M}\mathrm{C}\mathrm{A}(h_i^{(l-1)},{\mathcal{T}}_g)$$ 16

where h _i is the output of feature of node i at the (l – 1)­th layer. It should be noted that MCA takes nodes embeddings and coordinate embeddings as input in the current time step, when l = 1. ${\mathcal{T}}_g$ is a learned parameter used for aggregating the global features of the graph. As shown in Figure c, the self-attention layer has two attention layers to process global labels and node features, respectively. The first attention layer interacts with global labels with node features to generate induced features. The second attention layer uses these induced features to update the node features. In detail, the processes of this MCA module are as follows.

Step 1: Layer normalization for feature of nodes h ^(l–1) and ${\mathcal{T}}_g$ . In this step, the input node feature h _l and ${\mathcal{T}}_g$ are normalized first. Through the LN operator shown in eq , we can obtain features of nodes h _ln and q _ln.

$$h_{\ln }=\mathrm{LN}(h^l),{\mathcal{T}}_{\ln }=\mathrm{LN}({\mathcal{T}}_g^l)$$ 17

Step 2: Residual connection with the transformer encoder. This operator can be a formula as .

$$\{\begin{array}{l}{h}_{t1}=\mathrm{T}\mathrm{F}\mathrm{e}\mathrm{r}(h_{\ln },{\mathcal{T}}_{\ln })\\ h_{t2}=\mathrm{T}\mathrm{F}\mathrm{e}\mathrm{r}(h_{\ln },h_{t1})+h\end{array}$$ 18

where TFer (h _ln, q _ln) and TFer (h _ln, h _t1) are the standard transformer encoder with S = 8 multihead self-attention mechanism. For brevity, we only present the procedure of $\mathrm{T}\mathrm{F}\mathrm{e}\mathrm{r}(h_{\ln },{\mathcal{T}}_{\ln })$ . First, for the ith head, Q, K, and V are computed by linear transformation by .

$$\{\begin{array}{l}Q=W_Q·h_{\ln }\\ K=W_K·{\mathcal{T}}_g\\ V=W_V·{\mathcal{T}}_g\end{array}$$ 19

where W _Q , W _K , and W _V are the learnable matrixes. Then, the dot product between the query and the key is computed and scaled before applying the softmax function to obtain the attention weights. Thus, the output of the sth head can be obtained by

$$z_s=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\max \left(\frac{Q{K}^T}{\sqrt{d}}\right)·V$$ 20

Finally, the output of TFer (h _ln, h _t ) is obtained by a linear transformation.

$$z=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}(z_1,...,z_S)·W$$ 21

where z is the output of the transformer encoder, W is the linear transformer matrix, and z _s is the output of head s.

Step 3: Residual layer with FFN. Like , the feed-forward neural network is used to output the final node features.

$$\begin{array}{l}{h}_{\ln 1}=\mathrm{LN}(h_{t2})\\ h^l=h_{\ln 1}+\mathrm{F}\mathrm{F}\mathrm{N}(h_{\ln 1})\end{array}$$ 22

4.5. Scoring Block

The scoring block aims to estimate the gradient of the log density of atomic features and map the result into a score vector. As shown in Figure d, the scoring block consists of node MLP, pairwise block, distance MLP, and dist-transition block modules. Next, the score feature vectors are extracted by two aspects: node information and spatial information.

On the one hand, to estimate the gradient of the log density of atomic features, we utilize a MLP to transform the latent hidden vectors h _i , which are the outputs of the eEncoder.

$$S_{\theta }(A_i)={\mathrm{M}\mathrm{L}\mathrm{P}}_n(h_i)$$ 23

where MLP _n is a multilayer perceptron structured as (Linear, ReLU, Linear, ReLU, Linear).

On the other hand, for computing of gradient of pairwise distance, two nodes and edges are spliced according to the side information. Specifically, given an edge h _{e ij} , we can obtain the feature of node i and node j. Thus, we can extract the gradient of pairwise distance by a MLP.

$$s_{\theta }(d_{ij})={\mathrm{M}\mathrm{L}\mathrm{P}}_d(h_i\odot h_j\parallel h_{e_{ij}})$$ 24

where MLP _d is a multilayer perceptron structured as (Linear, ReLU, Linear, ReLU, Linear). ⊙ is an element-wise multiply operator, and ∥ is the concatenate operator. h _i and h _j are the features of nodes i and j, respectively. Second, the information on the score vector derived from pairwise distances and atomic coordinates R is subsequently integrated.

$$s_{\theta }(R_i)=\sum _{j\in N(i)}\frac{1}{d_{ij}}·s_{\theta }(d_{ij})·(r_i-r_j)$$ 25

In eq , since s _θ(d _ij ) depends only on d _ij , which is symmetrically invariant, s _θ(d _ij ) is translation invariant. Additionally, (r _i – r _j ) is the rotation and translation equivariance. Therefore, s _θ(R) has an equivariant property, thereby ensuring that the whole Markov chain process preserves rotation and translation invariance.

4.6. Loss Function

In general, the key to diffusion models is to train the reverse Markov chain to match the forward Markov chain. Therefore, the parameter θ needs to be adjusted so that the joint distribution of eq of the reverse Markov chain is close to that of eq of the forward process. This can be achieved by minimizing the Kullback–Leibler divergence between these two distributions

$$\begin{array}{l}\begin{array}{l}\mathrm{K}\mathrm{L}(q({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T)\parallel p_{\theta }({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T))\\ \stackrel{(i)}{=}-{\mathbb{E}}_{q({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T)}[\log p_{\theta }({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T)]\end{array}\\ \stackrel{(ii)}{=}\underset{≔-L_{\mathrm{V}\mathrm{L}\mathrm{B}}({\mathcal{M}}_0)}{\underbrace{{\mathbb{E}}_{q({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T)}[-\log p({\mathcal{M}}_T)-\sum _{t=1}^T\log \frac{p_{\theta }({\mathcal{M}}_{t-1}|{\mathcal{M}}_t)}{q({\mathcal{M}}_t|{\mathcal{M}}_{t-1})}]}}\\ \stackrel{(iii)}{\ge }\mathbb{E}[-\log p_{\theta }({\mathcal{M}}_0)]\end{array}$$ 26

where (i) is the definition of KL divergence. Both $q({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T)$ and $p_{\theta }({\mathcal{M}}_0,{\mathcal{M}}_1,...,{\mathcal{M}}_T)$ in (ii) are products of distributions. (iii) follows from Jensen’s inequality. In eq , (ii) is the variational lower bound (VLB) on the log-likelihood of the data ${\mathcal{M}}_0$ , which is the objective function for training a probabilistic generative model. Ho et al. proposed a reweighting term in L _VLB to improve the sample quality. Here, we use the following simplified training objective in reference to DDPM:

$${\mathcal{L}}_t={\mathbb{E}}_{{\mathcal{M}}_0}[\lambda (t){\sigma }_t^2{\parallel {\mathrm{\nabla }}_{{\mathcal{M}}_t}\log q({\mathcal{M}}_t|{\mathcal{M}}_0)-{\mathbf{s}}_{\theta }({\mathcal{M}}_t,t)\parallel }^2]$$ 27

The above formula is derived from reference, and we use it to denote a positive weighting function. We refer to PDMD in the training phase in order to ignore λ, λ is set to 1 to simplify the objective function. Such a simplified objective is equivalent to learning s_θ according to the gradient of the log density of the data distribution by sampling the diffusion molecules ${\mathcal{M}}_t$ at a random time step t. Alg. 1 gives out the pseudocode of the training process.

4.7. Sampling

In the sampling process, we employ Langevin dynamics to draw samples from the probability density. The detailed steps of the sampling process are as follows: first, the initial chaotic state ${\mathcal{M}}_t$ is sampled from the normal distribution $\mathbb{N}(0,I)$ . The Stein score μ_θ is then obtained through the DualESNet. Subsequently, according to $\mathbb{N}({\mathcal{M}}_T;{\mu }_{\theta },{\sigma }_T^{2}I)$ to obtain the chaotic state ${\mathcal{M}}_{T-1}$ at the next time step. By repeating this process T times, we finally generate the target molecule ${\mathcal{M}}_0$ . Alg. 2 is the pseudocode of the Sampling process.

5. Experiments

In this section, we present a discussion around the experiments of MGDM-Sa. We conducted an experiment to evaluate both molecular geometry generation and conditional molecular generation.

5.1. Data Set

Experiments for MGDM-Sa were performed on two public chemical spatial data sets: QM9. and GEOM-Drugs. The detailed information regarding the data set is presented below.

• GEOM is a benchmark data set for 3D molecule generation, containing millions of conformations of over 430k molecules, obtained from first-principles simulations. GEOM is split into two subsets: Geom-qm9 and Geom-Drugs. We use the GEOM-Drugs data set in this article. The GEOM-Drugs data set consists of medium-size drug molecules with a maximum of 181 atoms (91 heavy atoms) and contains an average of 44.2 atoms with an average of 6.5 rotatable bonds per molecule.

• QM9 is a benchmark data set for quantum property prediction, from the GDB17 data set. QM9 contains 133,885 small organic molecules consisting of no more than 9 heavy atoms (C, O, N, and F). Each molecule has 12 quantum properties such as geometry, energy, electron and thermodynamics, which are calculated by DFT.

5.2. Molecular Geometry Generation

In this subsection, we discuss the experiments of MGDM-Sa in molecular geometry generation from three aspects: baselines, metrics, and results.

5.3. Baselines

In this experiment, MGDM-Sa performed the geometric molecular generation task on the QM9 data set and GEOM-Drug data set. On the QM9 data set, we compare MGDM-Sa with ENF, SemlaFlow, EDM, MDM, JODO, 3D-EDiffMG, and G-SchNet. On the GEOM-Drug data set, we compare MGDM-Sa with ENF, EDM, G-SchNet, and MDM.

5.4. Metrics

We evaluate the quality of the generated molecular mass using four metrics:

• Validity: the percentage of molecules generated according to the chemical valence rule.

$$\mathcal{V}=\frac{N_v}{N_g}$$ 28

where $\mathcal{V}$ denotes the efficient rate, N _v is the number of effective molecules in the generated sample, and N _g is the number of generated molecules. The validity of the generated molecules was assessed using the RDKit chemistry toolkit. Molecules containing any atoms that violate standard chemical valence rules were deemed invalid.

• Uniqueness: the proportion of unique effective molecules in the generated molecular sample.

$$\mathcal{U}=\frac{N_{uv}}{N_g}$$ 29

where $\mathcal{U}$ denotes uniqueness, N _uv is the unique number of effective molecules in the generated sample, and N _g is the number of generated molecules.

• Novelty: the proportion of molecules not included in the training data set.

$$\mathcal{N}=\frac{N_n}{N_v}$$ 30

where $\mathcal{N}$ denotes the novelty, N _n is the number of generated novel molecules, that is, the number of molecules that are not in the training data set, and N _v denotes the effective number of molecules.

• Stability: the percentage of stable molecules in which all the atoms are in a stable state among the generated molecular samples.

$$\mathcal{S}=\frac{N_s}{N_v}$$ 31

where $\mathcal{S}$ denotes the molecular stability, N _s denotes the number of stable molecules, and N _v is the number of effective molecules.

5.5. Results

Tables and and Figures and present the performance comparison between MGDM-Sa and baseline models. As shown in Table , MGDM-Sa significantly outperforms existing baseline models on several key metrics of molecular generation. On the QM9 data set, in comparison with EDM, the MGDM-Sa exhibits significant enhancements in validity, stability, novelty, and uniqueness, with respective improvements of 8.27%, 7.16%, 27.9%, and 9.97%. Compared with MDM, MGDM-Sa achieves improvements of 1.3%, 6.36%, and 2.87% in effectiveness, stability, and uniqueness, respectively. MGDM-Sa has a higher effectiveness rate of 4.86% compared to JODO. Compared to 3D-EDiffMG-atten, MGDM-Sa demonstrates increases of 0.54%, 2.06%, and 5.07% in validity, stability, and uniqueness, respectively. Compared with SemlaFlow, MGDM-Sa has increased the effectiveness by 0.54% and the uniqueness by 2.07%. Additionally, MGDM-Sa-NA achieves performance comparable to that of 3D-EDiffMG-atten in terms of efficiency, stability, and uniqueness.

1. Comparison of MGDM-Sa and the Baseline Method in Three-Dimensional Molecular Generation Based on the QM9 Data Set .

methods	% $\mathcal{V}\uparrow$	% $\mathcal{U}\uparrow$	% $\mathcal{N}\uparrow$	% $\mathcal{S}\uparrow$
ENF	41.0	40.1	39.5	24.6
G-SchNet	85.9	80.9	57.6	85.6
EDM	91.7	90.5	59.9	91.1
MDM-NV	97.8	91.6	80.1	88.6
MDM	98.6	94.6	90.0	91.9
JODO	-	-	-	93.4
3D-EDiffMG	99.3	90.0	98.6	96.0
3D-EDiffMG-attn	99.4	92.4	99.3	96.2
SemlaFlow	99.4	95.4	-	99.7
MGDM-Sa-NA	98.71	90.38	87.42	96.7
MGDM-Sa	99.97	97.47	87.8	98.26

^aThis result is derived from comparing the performance of 10,000 molecules generated by MGDM-Sa with that of the benchmark model in the molecular geometry generation task. An upward arrow (↑) denotes that higher values are indicative of superior performance. $\mathcal{V}$ : validity; $\mathcal{U}$ : uniqueness; $\mathcal{N}$ : novelty; $\mathcal{S}$ : stability.

2. Comparison of MGDM-Sa and the Baseline Method in Three-Dimensional Molecular Generation Based on the GEOM-Drug Data Set .

methods	% $\mathcal{V}\uparrow$	% $\mathcal{U}\uparrow$	% $\mathcal{N}\uparrow$	% $\mathcal{S}\uparrow$
ENF	7.68	5.19	5.16	0
G-SchNet	14.9	14.7	14.6	8.07
EDM	68.6	68.6	68.6	13.7
MDM-NV	99.8	99.5	99.5	42.3
MDM	99.5	99.0	90.0	62.2
SemlaFlow	93.9	100.0	99.6	97.3
MGDM-Sa-NA	98.67	98.43	99.79	83.39
MGDM-Sa	97.04	97.04	99.98	89.98

^aThis result is derived from comparing the performance of 10,000 molecules generated by MGDM-Sa with that of the benchmark model in the molecular geometry generation task. An upward arrow (↑) denotes that higher values are indicative of superior performance. $\mathcal{V}$ : validity; $\mathcal{U}$ : uniqueness; $\mathcal{N}$ : novelty; $\mathcal{S}$ : stability.

2.

On the QM9 data set, the performance comparison between the MGDM-Sa and the benchmark models was conducted for the geometric molecule generation task. Orange indicates the most outstanding method.

3.

On the GEOM-Drug data set, the performance comparison between the MGDM-Sa and the benchmark models was conducted for the geometric molecule generation task. The orange color represents the method that demonstrated the most outstanding performance.

On the GEOM-Drug data set, in comparison with EDM, the MGDM-Sa exhibits significant enhancements in validity, stability, novelty, and uniqueness, with respective improvements of 28.44%, 28.44%, 31.38%, and 76.28%. Compared with MDM, MGDM-Sa achieves improvements of 27.78%, and 9.98% in stability and uniqueness, respectively. Meanwhile, MGDM-Sa-NA achieves comparable performance to 3D-EDiffMG-atten in terms of validity and uniqueness. The experimental results show that in the GEOM-Drug data set, MGDM-Sa has achieved a significant improvement in stability performance.

These results indicate that MGDM-Sa possesses significant advantages in the generation of high-quality molecular conformations. The essence of the performance improvement lies in the refined modeling of molecular interactions achieved by the MGDM-Sa. Specifically, DualESNet captures the differential characteristics of short-range strong interactions and long-range weak interactions, while effectively addressing the significant disparities in strength and range between these two types of interactions. In addition, MGDM-Sa follows the principle of equivariance of the molecular geometry to ensure the physical consistency of the generated conformations under translation and rotation operations. eEncoder enhances the representation capability of the relationship between nodes by capturing and integrating global and local information, so as to significantly improve the chemical rationality and structural diversity of the generated molecules.

On the other hand, we conducted ablation experiments to demonstrate the impact of attention in eEncoder on the performance of MGDM-Sa. We removed the self-attention layer in eEncoder as a variant of MGDM-Sa and named it MGDM-Sa-NA. The experimental results show that MGDM-Sa outperforms its variant MGDM-Sa-NA in terms of the validity, uniqueness, and stability of the generated molecules. From this, we can conclude that eEncoder can more comprehensively capture the relationships between nodes, thereby improving the accuracy and rationality of the generated molecules and enhancing the model’s expressive power. These results indicate that MGDM-Sa has a significant advantage in generating high-quality molecular conformations.

6. Conditional Molecular Generation

Generating molecules that satisfy certain desired properties is of positive significance for us to obtain drugs with specific properties. Next, we discuss the experiments of MGDM-Sa in conditional molecular generation from three aspects: metrics, baselines, and results.

6.1. Metrics

There are properties, that are often conditioned are

• α: Polarizability in cubic Bohr radius (Bohr³).

• μ: Dipole moment invariant (D).

• $\mathit{ϵ}$ _HOMO: The highest occupied molecular orbital energy in millielectron volts (meV).

• $\mathit{ϵ}$ _LUMO: The lowest unoccupied molecular orbital energy in millielectron volts (meV).

• ϵ_gap: The difference between ϵ_HOMO and ϵ_LUMO.

• C _v : The resulting molecules do not include the percentage of ions.

6.2. Experimental Setup

MGDM-Sa introduces the above six attributes to condition the generated molecules. Based on previous work EMD, we utilize the attribute classifier in EGNN. The set of QM9 is divided into two parts, each containing 50K samples. One-half denoted as D _c is used for classifier training, and the other half denoted as D _g is used for generative model training. The classifier ϕ _c then evaluates the attribute-based conditional generation samples by the mean absolute error between the predicted attribute value and the true attribute value.

6.3. Baselines

Here, we compare with several baseline references:

• Naive (Upper-Bond): The relationship between molecule and attribute is removed. The D _g is evaluated by the classifier ϕ _c . The labels are shuffled in D _g to predict molecules whose molecular properties require properties.

• Atoms: The classifier ϕ _c predicts the molecular properties on D _g by the number of atoms.

• EDM: The first diffusion model uses EGNN to generate 3D molecules.

• MDM: A diffusion model employs SchNet for the generation of 3D molecular structures, introduces separate equivariant encoders to capture interatomic forces of varying strengths, and incorporates a distribution control variable at each diffusion-inversion step to enhance molecular diversity.

• QM9 (Lower-Bond): The classifier ϕ _c evaluates the original D _c to predict molecular properties.

6.4. Results

The results of conditional molecule generation are shown in Table and Figure . First, MGDM-Sa performs well in predicting electronic structures: the mean absolute errors of ϵ_HOMO and ϵ_LUMO are 0.0049 and 0.0151 lower than those of MDM, respectively, and even outperform the QM9 benchmark. Second, the experimental results of MGDM-Sa on ϵ_gap are 0.0145 lower than those of the comparison model MDM and 0.0345 lower than the QM9 benchmark. Meanwhile, in terms of μ, MGDM-Sa outperforms EDM and MDM, with the mean absolute error of MGDM-Sa being 0.0061 lower than that of EDM. However, the results of MGDM-Sa on C _v and α are not ideal. This might be due to the fact that MGDM-Sa may overfit the electronic property targets during the generation process, sacrificing the rationality and diversity of molecular structures. Particularly, the lower index results compared to the QM9 benchmark might be attributed to the smaller distribution difference between D _c and D _g . The experimental results indicate that MGDM-Sa has the ability to generate molecules with the targeted properties.

3. A Performance Comparison of the MGDM-Sa against Baseline Models in the Conditional Molecule Generation Task .

methods	α ↓	μ ↓	ϵHOMO ↓	ϵLUMO ↓	ϵgap ↓	C v ↓
naive (U)	9.013	1.616	0.645	1.457	1.472	6.857
#atoms	3.862	1.053	0.426	0.813	0.866	1.971
EDM	2.760	1.111	0.356	0.584	0.655	1.101
MDM	1.591	1.177	0.019	0.040	0.044	1.647
MGDM-Sa	3.0811	1.1049	0.0141	0.0249	0.0295	5.2179
QM9 (L)	0.100	0.043	0.039	0.036	0.064	0.040

^aThe downward arrow (↓) signifies that lower values correspond to improved model performance.

4.

Comparison of various methods in the conditional molecule generation task. Orange represents MGMD-Sa.

7. Analysis

7.1. Statistical Analysis

In this subsection, we will analyze the molecules generated by MGDM-Sa from three perspectives: the distribution of molecular properties, the distribution of data types, and the comparison of molecular structures.

7.2. Distribution of Molecular Properties

To validate that MGDM-Sa is capable of generating molecules with reasonable physicochemical and pharmacokinetic properties, we conducted an analysis of the distribution of 10,000 molecular samples generated by MGDM-Sa across five key properties, essential for drug development. Figures and , respectively, illustrate the distribution of five properties for 10,000 samples generated by MGDM-Sa on the QM9 data set and GEOM-Drug. The specific analysis is as follows:

• log P: The median values of log P on the QM9 data set and the GEOM-Drug data set are 1.17 and 2.82, respectively. Lipinski’s five rules suggest that log P should be less than or equal to 5. This suggests that these molecules maintain a favorable balance in terms of hydrophilicity.

• SAScore: The median values of SAScore in the QM9 data set and the GEOM-Drug data set are 3.87 and 4.02, respectively. Moreover, the IQRs are (3.2, 4.5) and (3.5, 4.5), respectively. This suggests that the synthetic accessibility of these molecules is moderate, indicating that they can be synthesized through feasible routes without introducing excessive complexity during the manufacturing process.

• TPSA: The molecules in the QM9 data set generally exhibit relatively low TPSA values, as they are predominantly small and low-polarity compounds. These molecules typically lack polar functional groups or large structural frameworks, which results in the low TPSA values observed in the molecules generated by MGDM-Sa. The median TPSA was 12.47 Ų, with an IQR of 3.0 Ų to 23 Ų. The low value indicates that most of these molecules are relatively hydrophobic, have strong lipophilicity, and have a good ability to penetrate cell membranes. On the GEOM-Drug data set, the median value of TPSA is 61.44 Ų. It is indicated that the molecule as a whole has a certain degree of polarity. Furthermore, the IQR range is wide, indicating that the polarity distribution of the molecules is broad, possibly encompassing various structures ranging from polar to nonpolar. This also indicates the complexity and diversity of the molecular structures in the GEOM-Drug data set.

• MolWt: On the QM9 data set, the median MolWt is 110.20 Da. This complies with Lipinski’s five rules, as typically an oral drug molecule should have a molecular weight of less than 500 Da. Additionally, in the GEOM-Drug data set, the median MolWt is 337.460 Da, which falls within the ideal range of (200,500) Da. Moreover, the IQR shows that most molecules are concentrated in the range of (280,390) Da, which meets the drug-likeness requirements.

• QED: Although the QED value on the QM9 data set is slightly lower at 0.46, the median QED on the GEOM-Drug data set reaches 0.61. This indicates that these compounds exhibit good drug similarity and can be regarded as promising candidates for drug development.

5.

Distribution of 10,000 molecules generated by MGDM-Sa on the QM9 data set in terms of log P, SA score, TPSA, MolWt, and QED.

6.

Distribution of 10,000 molecules generated by MGDM-Sa on the GEOM-Drug data set in terms of log P, SA score, TPSA, MolWt, and QED.

7.3. Comparison of Molecular Structures

We compared the three-dimensional structures of the molecules in the QM9 data set with those generated by MGDM-Sa and also compared the molecules in the GEOM-Drug data set with those generated by MGDM-Sa. For example, Figure a reveals that the difference in bond angle at the same position of the two molecules is merely 0.9d°, the difference in bond length is only 0.1 Å, and the difference in the dihedral angle is 44.8°. The six comparison diagrams presented in Figure demonstrate that there are minor differences in bond length and bond angle between the molecules of QM9 and MGDM-Sa. However, a certain disparity exists in the dihedral angle. Therefore, it can be evidenced that MGDM-Sa has the capacity to learn molecular topology and is capable of generating drug-like molecules that are both similar to and novel compared to those in the QM9 data set or GEOM-Drug data set.

7.

Light blue represents the molecule generated by MGDM-Sa, while green represents the molecules in the original data set. (a,c,e) Molecules formed by the interaction of the molecules in QM9 with MGDM-Sa. (b,d,f) Molecules generated by the interaction of the molecules in GEOM-Drug with MGDM-Sa. Among them, the purple ones are bond angles, the blue ones are dihedral angles, and the red ones are bond lengths.

7.4. Molecular Property Analysis

In this subsection, to verify the potential of the molecules generated by MGDM-Sa to become drug candidates, we will discuss and analyze the new molecules from three aspects: basic molecular properties, ADMET predictions, and molecular docking.

First, the molecules generated by MGDM-Sa have the potential to be viable drug candidates. We randomly selected four molecules from the samples, which were generated by MGDM-Sa. These molecular structures are shown in Figure . Furthermore, Table presents the prediction results of the nine properties. For instance, the log P of M-1 is 2.7, which falls within the reasonable range of (1.5). The QED of M-2 is 0.8. A QED greater than 0.67 is generally regarded as having good drug potential. The SAScores of the four molecules are all less than 5, which indicates that they are relatively easy-to-synthesize compounds.

8.

(a,b) Structural fragments generated by MGDM-Sa in the QM9 data set. (c,d) Molecular structures generated by MGDM-Sa in the GEOM-Drug data set.

4. Prediction of Nine Fundamental Properties of Some Generated Molecules .

^aLog P: logarithm of partition coefficient; QED: quantitative score of drug similarity; SAScore: composite feasibility score; MolWt: molecular weight; TPSA: topological polar surface area; HBA: number of hydrogen bond acceptors; HBD: number of hydrogen bond donors; rotatable bonds: number of rotatable keys; aromatic rings: number of aromatic rings.M1 and M2 are the molecules generated by MGDM-Sa on the QM9 data set. M3 and M4 are the molecules generated by MGDM-Sa on the GEOM-Drug data set.

Second, to verify the capability of MGDM-Sa to generate drug-like molecules, we conducted an analysis of the absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of the generated molecules. As shown in Table , the compounds are mostly within a reasonable range in drug development, meeting the common standards for drug candidates. As shown in Table , most of these properties fall within the reasonable range for drug development and meet the common standards for drug candidates. For instance, the Coca-2 of M-1 is −4.67, which is greater than −1.15. The PPB of M-2 is 47.892%, which is less than 90%. The CYD2D6-I of M-3 is 0, indicating it belongs to the noninhibitor category. The DILI of M-4 is 0, suggesting that it is a drug without DILI risk. Moreover, combining Tables and , we can see that the molecules generated by MGDM-Sa perform well in terms of physical and chemical properties, drug-like properties, absorption, distribution, and toxicity. Therefore, MGDM-Sa can generate molecules with a potential druggability.

5. Prediction of Molecular Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET Properties,

molecules	Caco-2	HIA	PPB	BBB	VDss	CYP 2D6-I	CYP 3A4-I	CYP 2C9-I	CYP 2D6-S	CYP 3A4-S	CYP 2C9-S	CYP 2C19-I	CYP 2C19-S	CYP 2B6-I	CYP 2B6-S	CL	T 1/2	hERG	AMES	DILI
M-1	–4.67	1	38.174	1	0.361	0	0	0	0	0	1	0	0	0	1	12.028	1.718	0	0	0
M-2	–4.323	1	47.892	1	0.412	0	0	0	1	0	0	0	1	0	1	11.074	1.286	0	0	0
M-3	–5.182	1	53.633	1	–0.035	0	0	0	0	0	0	1	0	0	0	6.771	0.77	0	0	0
M-4	–4.959	1	59.046	1	0.242	0	0	0	0	1	0	0	1	1	0	8.97	1.623	0	0	0

^aIn the table, a value of 0 indicates no activity and 1 indicates significant activity.

^bCaco-2/Caco-2 permeability; HIA: human intestinal absorption; PPB: plasma protein binding; BBB: plasma protein binding; VDss: volume distribution; CYP/CYP450 enzyme; CL: clearance; T _1/2: half-life; DLI: drug induced liver injury. M1 and M2 are the molecules generated by MGDM-Sa on the QM9 data set. M3 and M4 are the molecules generated by MGDM-Sa on the GEOM-Drug data set.

Finally, to evaluate the ability of our model to generate bioactive molecules, two popular targets, hematopoietic progenitor kinase 1 (HPK1, PDB ID: 7M0M) and leucine-rich repeat kinase 2 (LRRK2, PDB ID: 8FO7), were used as examples to study the docking of the generated molecules with the targets. As shown in Figure , the case study reveals that the molecules generated via MGDM-Sa are not only biologically active but also exhibit drug-like properties, structural diversity, and novelty. This underscores the capability of our approach to generate high-quality candidates with desired characteristics, making it a valuable tool for de novo drug design.

9.

Molecular docking structure; yellow dashed lines indicate hydrogen bonds. (a) Shows the docking of the molecules generated by MGDM-Sa on the QM9 data set with HPK1. (b) Shows the docking of the molecules generated by MGDM-Sa on the GEOM-Drug data set with LRRK2.

8. Conclusion

In this article, we propose a novel diffusion model MGDM-Sa for 3D molecular generation tasks, which combines a biequivariant fractional neural network called DualESNet. The DualESNet models the bond lengths of different interatomic forces to capture complex interatomic interaction relationships and enhances the ability of the model to characterize atomic connectivity. In particular, in DualESNet, we design a new equivariant encoder, eEncoder, which integrates the EGNN and self-attention. eEncoder effectively integrates global and local information to comprehensively capture complex relationships and dependencies in the molecular graph, thereby enhancing the expression ability of the model.

We performed both geometric molecule generation and conditional molecule generation experiments. Experimental results show that MGDM-Sa can generate effective and stable molecules. By comparing MGDM-Sa with the baseline model, it is proven that MGDM-Sa has advantages in both geometric molecular generation performance and generation of molecules with expected properties.

We performed physicochemical and medicinal chemical property prediction, ADMET prediction, and molecular docking on the molecules generated by MGDM-Sa. The results show that the molecules generated based on MGDM-Sa not only have good biological activities but also maintain good drug-like properties. We demonstrate that MGDM-Sa is capable of generating effective candidates and will be a useful de novo drug design tool.

The data sets and codes used in this study are available at GitHub: https://github.com/yha2333/MGDM.git.

The authors declare no competing financial interest.