LigGen—a GEN-AI based ligand generation approach for de-novo drug design

Abstract

Navigating the vast chemical space to identify potent therapeutic agents with optimal pharmacokinetic properties remains a formidable bottleneck in pharmaceutical development. While virtual screening is limited to the exploration of existing chemical libraries, de novo design offers a complementary approach by constructing novel chemical entities “from scratch,” potentially accessing unexplored regions of the chemical universe. De novo design may lead to novel scaffolds or compounds when compared to searching the compounds from chemical libraries. This study introduces LigGen, a novel tool for de novo drug design (DNDD) that combines fragment-based drug design (FBDD) with recurrent neural networks (RNNs) and Monte Carlo Simulated Annealing (MCSA) to generate ligands for target proteins. LigGen is effective in generating novel and synthesizeable high-affinity ligands for various biological drug-targets.  The efficacy of the tool is demonstrated by testing it on CrossDocked dataset, as well as by generating ligands for proteins Beta-secretase 1 (BACE1), Monoamine Oxidase B (MAO-B), and Acetylcholinesterase (AChE). LigGen presents a promising approach for advancing drug discovery, offering a robust platform for novel drug design.

Introduction

The pharmaceutical industry faces significant challenges in the discovery and development of novel therapeutic agents. One of the major obstacles in the drug design and development process was the time and financial investment required for the creation of novel therapeutic agents¹. It is estimated that, of 10000 drug candidates, only 5 will advance from preclinical testing to human testing, and of those, only 1 will reach the market². It is well-known that in the hunt for new novel drug candidates, a very small portion of the chemical space has been explored. Because the chemical space is so enormous, with certain chemical libraries³ having compounds in the range , it is not feasible to screen all the compounds in these libraries. Instead, designing novel drug candidates using atom-by-atom construction or from fragments can be a pragmatic approach. Consequently, proposing molecules with novel scaffolds and synthesizing them for inclusion into drug development is a challenging problem for medicinal and organic chemists². Researchers all across the world turned to computational methods like virtual screening (VS), molecular docking (MD), de-novo drug design (DNDD), and artificial intelligence (AI) to reduce these difficulties and barriers.

One may broadly categorise the current approaches used in structure-based drug design into two groups. Virtual screening (VS), also known as “finding” ligands for a certain receptor, is the focus of the first category. In VS, a vast number of molecules are screened to identify those that bind effectively to the receptor’s binding pocket. Virtual screening’s primary benefit is that it reduces the amount of synthetic work required to find novel lead compounds⁴. Virtual screening offers many benefits, but it still takes a lot of time since it needs to filter through a lot of compounds to find the right compound for a particular receptor. De Novo Drug Design is another category of structure-based drug design as it focuses on “building” the ligands. The technique of developing novel drug compounds with computational growth methods that meet a predetermined set of requirements is known as de novo drug design (DNDD)⁵. “De novo” means “from the beginning,” highlighting the fact that this process may create new molecular entities without requiring a starting template².

A dominant strategy within DNDD is the gradual assembly of tiny components into larger ligand molecules through the use of fragment-based drug design (FBDD). These components can be fragments or atoms. Genetic algorithms are often utilized to build ligands. The main benefit of this approach is that new structures that aren’t in any database may be proposed⁴. For the purpose of creating ligands for various proteins, fragment-based drug design has been widely utilised by both industry and academia in recent years⁶. Fragments are essentially low-weight, tiny organic molecules that satisfy the “rule of three”: molecular weight (MW) < 300, ClogP 3, and a number of hydrogen bond acceptors 3⁷. The rationale behind FBDD lies in its ability to probe a vast chemical space using a relatively small library; since the number of possible small fragments is manageable compared to the astronomical number of drug-sized molecules, combining these fragments allows for a more efficient exploration of chemical diversity. Furthermore, FBDD emphasizes Ligand Efficiency (LE), a metric that normalizes binding affinity by the number of heavy atoms. High LE indicates that each atom in the fragment contributes significantly to binding, ensuring that the final molecule maintains high quality interactions as it grows in size. Consequently, FBDD offers benefits such as higher hit rates, enhanced physicochemical properties, and a higher potential for chemical uniqueness⁸.

A number of tools exist for de novo design. Among them, the program LUDI⁹ operates by using collections of fragments that can be freely positioned within designated subpockets termed “interaction sites.” These fragments are subsequently linked together through spacers or linkers to form viable, larger molecular structures. ilib diverse¹⁰ is a tool used for cataloging ‘drug-like’ compound libraries starting from a well-balanced molecular fragment set with high druglikeness¹¹. CombiSmog¹² utilizes a knowledge-based potential that is derived from a collection of 1,000 protein-ligand complexes. Additionally, it employs a Monte Carlo ligand growth algorithm in order to build new and unique ligands. LigBuilder V3¹³ uses genetic algorithm (GA) for the creation of ligands and offers functionalities like Multi-Target Drug Design and Fragment-Based Drug Design. The successful use of de novo design methodologies has been documented in numerous published examples, which include the discovery of ligands of FK506 binding protein-12¹⁴, DNA gyrase¹⁵, thrombin employing LUDI¹⁶, and dihydroorotate dehydrogenase using Sprout¹⁷.

Another popular technique for de-novo design includes ligand generation using deep neural networks such as RNNs. Natural language processing and translation are among the machine learning tasks that Recurrent Neural Networks (RNNs) are very good at solving. While early Neural network-based RNNs have been utilised to predict properties like aqueous solubility¹⁸, recent focus has shifted to generation. Preferred areas of chemical space were sampled using RNNs acting as autoencoders to create a latent representation of molecular structure¹⁹. Crucially, a number of research teams have shown that RNNs can be used to create canonical SMILES strings and that transfer learning can be used to improve them²⁰. AstraZeneca researchers have expanded SMILES-generating RNNs to reinforcement learning by using this idea. Using this method, they were able to produce sets of structures with high target activity and other desired characteristics²¹. Pocket2Mol²² is a popular deep learning-based tool for ligand generation. It is a generative model that designs molecules by explicitly considering the three-dimensional shape and chemical properties of a protein’s binding pocket.

By combining RNNs with traditional computer techniques, researchers can increase the accuracy and efficiency of the drug discovery process. By combining the most beneficial features of both strategies, this combination provides a more robust platform for the development of novel drugs. While emerging deep learning approaches such as diffusion and flow-matching models offer geometric generation capabilities, they often operate as “black boxes” regarding the chemical logic of assembly. In contrast, the tool proposed in this study, LigGen, utilizes a transparent, iterative growth process that mimics the physical addition of substructures, offering greater interpretability in the design pathway. This paper introduces LigGen, a novel program for structure-based de novo drug design that addresses the critical need for generating synthesizable lead compounds with high binding affinity. LigGen leverages the strengths of both recurrent neural networks (RNNs) for intelligent fragment selection and fragment-based drug design (FBDD) principles to iteratively construct ligands within the target protein’s binding pocket. Crucially, LigGen employs a Monte Carlo Simulated Annealing algorithm for ligand growth, allowing for efficient exploration of conformational space and optimization of binding interactions. LigGen integrates a synthesizability score into its optimization process, ensuring that the generated ligands are not only predicted to bind effectively but are also chemically tractable/synthesizeable. This multi-objective optimization, combining binding affinity (assessed using Autodock-Vina) and synthesizability, represents a significant improvement over existing methods. On the fly generation of the fragment to grow ligands through RNN-GRU architecture is another significant difference when compared to existing de novo design tools.

Methods

Monte Carlo simulated annealing

Monte Carlo simulated annealing (MCSA) is a stochastic optimization technique inspired by the annealing process in metallurgy and material science. MCSA is a standard, well-established technique and is widely used to identify the global minimum²³. Similar to how slowly cooling a metal or material allows its atoms to arrange themselves in a low/least-energy, stable configuration, MCSA probabilistically explores solution space to find the optimal solution for a given problem²⁴. This approach is particularly valuable for tackling complex problems with many local minima, where traditional optimization methods might fail to find optimal solution corresponding to global minimum. 

The MCSA algorithm initiates with an empty ligand state (a ligand with no atoms). In each subsequent iteration, a new neighbouring ligand state is generated by proposing a modification to the current ligand. This modification involves adding a chemically valid fragment to the existing ligand structure. The “energy” of a given ligand state is defined as a weighted sum of two critical scores: its binding affinity, as estimated by the Autodock Vina, and its synthesizability score, derived from RDKit’s SAscore.

This objective function is designed to operate as a dynamic trade-off rather than a rigidly normalized equation. By allowing the raw Vina score (typically negative kcal/mol) and the SA score (positive penalty) to interact directly, the algorithm naturally favors larger, high-affinity binders only when the gain in binding energy sufficiently outweighs the penalty incurred by increased synthetic complexity.

• If is negative (indicating an improvement in the combined score), the new ligand state is always accepted.

• If is positive (indicating a worsening in the combined score), the new state is accepted probabilistically using the Metropolis Acceptance Criterion:

  

  where T is the current “temperature” of the system.

Note that unlike the standard Boltzmann distribution in statistical mechanics, physical constants such as the Boltzmann constant () or the ideal gas constant (R) are omitted from this equation, as they are mathematically absorbed into the definition of the algorithmic temperature itself.

The “temperature” is a control parameter that starts at a high value, promoting broad exploration of the chemical space and acceptance of even unfavorable moves. As the simulation progresses, the temperature is gradually lowered by a factor of alpha after each accepted modification. This annealing schedule progressively reduces the probability of accepting energetically unfavorable states, allowing the system to converge towards optimal or near-optimal ligand structures. This controlled decrease in temperature ensures that the algorithm shifts from an exploratory phase to a refinement phase, guiding the ligand generation towards high-quality solutions.

Recurrent neural network

Recurrent Neural Networks (RNNs) are a special type of neural network that’s good at handling data that comes in a specific order, like words, sentences, paragraphs or music. They do this by having loops in their connections, allowing them to remember what they have seen before. At each step, it combines the new information it receives with what it learned from the previous step using the same underlying parameters (Fig. 1).

Fig. 1.

Simplified data flow in the RNN-GRU model. Each input SELFIES token is converted to an embedding, processed sequentially by GRU layers, and finally mapped to a probability distribution for the next token.

Training a Recurrent Neural Network (RNN) for sequence modelling generally involves using maximum likelihood estimation to predict the next token in the target sequence based on the tokens from previous steps. At each step, the model generates a probability distribution for the next likely character, with the goal of maximizing the probability assigned to the correct token²¹:

The recurrent neural network (RNN) model used in LigGen is based on a Gated Recurrent Unit (GRU) architecture, implemented using PyTorch. The model operates over SELFIES (Self-Referencing Embedded Strings)²⁵ tokens, which are robust, simple, string-based molecular representation. Unlike SMILES-based models which require complex grammar masking to ensure validity, the SELFIES representation guarantees that every generated token sequence corresponds to a valid molecular graph. This is critical in generative models to avoid chemically invalid outputs, which are more frequent when using SMILES²⁵. During inference, the model first generates SELFIES, which are then automatically converted to SMILES before being passed to LigGen for ligand generation.The architecture comprises of Embedding layer (256 dimensions) converting token indices to dense vectors, Stacked GRU layers (3 layers, 512 hidden units), and finally a linear output layer mapping to token probabilities. Our generative model employs a Gated Recurrent Unit (GRU) architecture. We selected GRU over Long Short-Term Memory (LSTM) units because they offer a favourable balance between performance and computational efficiency, often achieving comparable results in sequence modelling tasks while using fewer parameters²⁶. Given that fragments are used as modular building blocks rather than full-molecule templates, the GRU architecture efficiently captures the local substructural syntax required for valid fragment generation without the computational overhead of Transformer-based models.

Once a recurrent neural network (RNN) is trained on target fragments represented as SELFIES strings, it can generate new fragments based on the conditional probability distributions it learned during training. The process begins with a start token as the initial input. At each subsequent timestep, an output token is sampled from the predicted probability distribution P() over the vocabulary X. This sampled token is then used as the input for the next timestep. The sequence generation continues until an end token is sampled, signalling the completion of the sequence²¹. Using an RNN for fragment generation offers distinct advantages over static fragment libraries. Unlike a fixed database where the chemical space is discrete and limited to enumerated entries, the generative model learns a probability distribution over the chemical space. This allows for interpolation, enabling the generation of chemically valid fragments that may not be explicitly present in the training set, thereby expanding the potential diversity of building blocks beyond a fixed list. However, it is essential to note that the generative capabilities remain influenced by the underlying distribution of the training data. Additionally, the generative approach offers computational efficiency by eliminating the need to store or query massive fragment libraries in memory during ligand growth.

3d Structure formation and scoring

LigGen generates a single 3D conformer using ETKDG embedding algorithm²⁷, during this growth phase for computational efficiency. Since the MCSA protocol requires thousands of iterations, generating and docking multiple conformers at every step would be computationally prohibitive. However, the subsequent local optimization step (BFGS) allows the ligand to adjust its torsion angles within the pocket, effectively exploring the local conformational space without the need for an exhaustive ensemble search. RDKit stands out as a reliable open-source toolkit widely used in cheminformatics for a broad spectrum of tasks, including molecular modelling, computational chemistry, and machine learning.

Additionally, RDKit includes a tool for calculating the Synthetic Accessibility (SA) score, which estimates the ease of synthesizing a given molecule in the laboratory. RDKit sascorer package is used for calculating Synthesizability score of ligands. The SA score considers factors such as molecular complexity, the presence of specific functional groups, and overall structural features. The synthetic accessibility score is calculated as a combination of two components:

The fragmentScore captures the “historical synthetic knowledge” by analyzing common structural features in a large number of already synthesized molecules. It is the mean of contributions of all fragments in the molecule²⁸. The complexityScore is a numerical value that indicates whether or not the molecules have complex structural properties. It is determined by combining the sizePenalty, macrocyclePenalty, stereoComplexityScore, and ringComplexityScore²⁸.

AutoDock Vina is a widely used software for molecular docking that predicts the preferred orientation of a small molecule (ligand) when bound to a protein (receptor), aiding in the assessment of the molecule’s binding affinity²⁹. The scoring function in AutoDock Vina evaluates different poses of the ligand by estimating their binding affinities. This is achieved through a combination of empirical and knowledge-based approaches that consider factors such as hydrogen bonding, hydrophobic interactions, electrostatic interactions and spatial complementarity between the ligand and the receptor²⁹. 3d conformer is used for the calculation of binding affinity by Autodock vina and synthesizability score by Rdkit. The weighted sum of these two is taken in the Metropolis acceptance criterion, which governs the acceptance or rejection of the newly formed ligand. The aim of this acceptance criteria is such that it tries to decrease the binding affinity score and increase the synthesizability score of the ligand.

BFGS

The Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm is a widely used iterative technique for addressing unconstrained nonlinear optimization problems³⁰. It is a member of the quasi-Newton technique family, which aims to approximate the Newton-Raphson method without requiring the second derivatives to be calculated³¹. By using an estimate of the objective function’s Hessian matrix, BFGS incorporates curvature information to identify the direction of descent.³². This approximation is iteratively improved using gradient evaluations through a generalized secant method. This approach is advantageous compared to Newton’s method, as it avoids the need to compute the exact Hessian at each step, striking a balance between convergence speed and computational efficiency³². This algorithm is widely utilized in many domains, including machine learning, computational chemistry, and structural optimization, due to its fast handling of large-scale optimization problems. Numerous real-world applications have proven to its reliability and efficacy, and many optimization software packages use it. One such popular software using the BFGS algorithm for minimisation is AutoDock-Vina, it utilizes the BFGS algorithm to refine the ligand conformation and achieve a more accurate prediction of binding modes and affinities.

Fragments to ligands

LigGen starts by taking SMILES of fragments and grid box details like size and centre (six variables). Grid box is essential for calculating the Autodock-vina score and locally optimizing the pose of the generated ligand. The program starts the building of the ligand from user-given coordinates which are inside the grid box. Input SMILES to LigGen can be generated using the RNN network or can be from an already available fragment library.

During each iteration program selects a random fragment and attaches it to the ligand that is computed thus far. This stochastic selection process is a deliberate design choice intended to maximize the diversity of the generated chemical space. By utilizing different random seeds for each generation run, LigGen prevents mode collapse—a common issue in deterministic optimizers—ensuring that the algorithm explores a broad range of topological solutions rather than repeatedly converging on a single, identical local minimum. To attach the chosen fragment to the ligand, the program stochastically selects an attachment point within the ligand’s SELFIES string. Rather than restricting growth to pre-defined reaction templates (e.g., BRICS or RECAP rules) which limits the explore-able chemical space to known reaction pathways, this stochastic approach allows for the exploration of novel topological connections. To ensure chemical validity despite the absence of rigid templates, the algorithm employs a strict sanitization protocol. Immediately after a stochastic attachment is proposed, the resulting molecule is validated using RDKit’s sanitization function. Any attachment that results in a chemically invalid graph (e.g., valence violations or aromaticity issues) is automatically rejected, triggering a new attachment attempt.The SMILES of the new ligand undergo 3D structure formation. Since the coordinates of this new ligand are entirely different from the previous one, the structure of the new ligand is aligned with the previous one. This alignment utilizes the Maximum Common Substructure (MCS) to constrain the conformation of the parent structure, ensuring that the new fragment grows into the pocket without disrupting the established binding mode of the previous iteration. Even after the alignment, the position may not be optimal (having the least Autodock-vina score), so local energy minimization of the ligand is performed using Autodock-vina, which utilizes the BFGS algorithm to find the conformer providing the least binding affinity.

Acceptance and rejection of a state are carried out using Metropolis acceptance criteria, which depend on the total score and current temperature. The total score comprises the weighted sum of the Autodock-vina score and synthesizability score given by RDKit on this newly optimized ligand. Ligands that have a higher synthesizability score have a higher probability of being generated in the lab, while those with a lower synthesizability score carry a higher Autodock-vina score weight, thus offering better binding affinity. The temperature of the system decreases by a factor of alpha after every accepted state, where alpha ranges between 0 and 1, making it harder for the program to accept the worst state in the later stage of ligand generation. Mathematically, change in temperature can be expressed as:

Here, T is the initial temperature, and T’ is the temperature after the system has accepted a state, reflecting the gradual cooling process. The program keeps growing the ligand until it reaches the specified size. A flow chart illustrating the program is provided in Fig. 2.

Fig. 2.

Flow chart of the ligand generation program. It outlines the process from initial ligand generation through optimization and state acceptance/rejection based on the Metropolis acceptance criteria.

Results and discussion

In this section, we evaluate the performance of LigGen in comparison with various de novo drug design tools. Section “Crossdocked dataset” presents LigGen’s performance on the CrossDock dataset, while section “Comparative analysis of LigGen for alzheimer’s-related targets: BACE1, MAO-B, and AChE” discusses LigGen’s performance on three Alzheimer ’s-related proteins: BACE1, MAO-B, and AChE. The ligands generated using LigGen were constructed from fragments produced by LigGen’s recurrent neural network (RNN), which was initially trained on the ChEMBL dataset comprising approximately 0.5 million compounds to learn the grammar of SMILES strings. To make the model generate drug-relevant fragments, this RNN was fine-tuned using LigBuilder V3’s curated fragment library. This library includes fragments which are “Bio-Like” and “Drug-Like”, derived via retrosynthetic fragmentation (e.g., cleaving rotatable bonds while preserving functional groups) from sources such as ChEMBL, DrugBank³³, WDI³⁴, and SuperDrug³⁵. LigBuilder V3’s fragment dataset, which contained approximately 2,000 fragments, enables it to generate novel, diverse fragments while maintaining compatibility with known chemical scaffolds.

Crossdocked dataset

We evaluated LigGen using the test set of the CrossDocked dataset, originally used in the paper “A 3D Generative Model for Structure-Based Drug Design”³⁷. The test set contains 100 diverse protein-ligand pairs. Table 1 summarizes the molecular properties of the ligands generated for these targets. We compared LigGen against multiple baselines, including 3D-SBDD³⁷, Pocket2Mol²², GraphBP³⁸, TargetDiff³⁹, DecompDiff⁴⁰, DiffSBDD⁴¹, FLAG⁴², DrugGPS⁴³, Lingo3DMol⁴⁴, and Frag2Seq³⁶. Baseline results are sourced from the paper “Fragment and Geometry Aware Tokenization of Molecules for Structure-Based Drug Design Using Language Models”³⁶ .

Table 1.

Summary of molecular properties for generated molecules by our method and other baselines.

Methods	Vina score ()	QED ()	SA ()	Lipinski ()	Diversity ()	Time (s, )
Test set*	– 6.87 ± 2.32	0.47 ± 0.20	0.72 ± 0.14	4.34 ± 1.14
3D-SBDD*	– 5.88 ± 1.91	0.50 ± 0.17	0.67 ± 0.14	4.78 ± 0.51	0.74 ± 0.09	15986.4 ± 9851.0
Pocket2Mol*	– 7.05 ± 2.80	0.57 ± 0.16	0.75 ± 0.12	4.93 ± 0.27	0.73 ± 0.15	2827.3 ± 1456.8
GraphBP*	– 4.71 ± 4.03	0.50 ± 0.12	0.30 ± 0.09	4.88 ± 0.37	0.84 ± 0.01	1162.8 ± 438.5
TargetDiff*	– 7.31 ± 2.47	0.48 ± 0.20	0.58 ± 0.13	4.59 ± 0.83	0.71 ± 0.09	3428
DecompDiff*	– 6.60 ± 2.11	0.49 ± 0.21	0.65 ± 0.14	4.49 ± 1.02	0.72 ± 0.10	6189
DiffSBDD*	– 7.17 ± 3.28	0.55 ± 0.20	0.72 ± 0.12	4.74 ± 0.59	0.71 ± 0.07	629.9 ± 277.2
FLAG*	– 6.38 ± 3.24	0.48 ± 0.19	0.70 ± 0.15	4.65 ± 0.74	0.70 ± 0.14	1289.1 ± 378.0
DrugGPS*	– 6.60 ± 2.38	0.46 ± 0.21	0.62 ± 0.15	4.49 ± 0.99	0.73 ± 0.10	1007.8 ± 554.1
Lingo3DMol*	– 7.25 ± 1.69	0.26 ± 0.15	0.65 ± 0.08	3.12 ± 1.25	0.48 ± 0.12	1481.9 ± 1512.8
Frag2Seq*	– 7.36 ± 1.96	0.64 ± 0.15	0.64 ± 0.11	4.98 ± 0.11	0.71 ± 0.07	48.8 ± 14.6
LigGen	– 7.50 ± 1.83	0.43 ± 0.22	0.68 ± 0.10	4.60 ± 0.73	0.69 ± 0.13	1271.335 ± 3107.77

 The best values in each column are shown in [bold].

* denotes the property results taken from the Frag2Seq³⁶ paper.

The binding affinity between generated molecules and given protein pockets is estimated by Vina Score; QED is a metric that evaluates a molecule’s drug-likeness based on its physicochemical properties; SA (Synthetic Accessibility) calculates the ease of synthesis. Note that for the CrossDocked comparative benchmark, we report the normalized SA score (range 0-1, higher is better) to facilitate comparison with baseline methods. This is calculated as , where a value of 1.0 corresponds to the best possible raw score of 1, and 0.0 corresponds to the worst raw score of 10; Lipinski assesses a molecule’s drug-likeness by determining how well it complies with Lipinski’s rule of five; diversity estimates the average pairwise diversity of molecules produced for a binding pocket, which is determined by 1-Tanimotosimilarity; Time is the average amount of time needed to produce 100 molecules for a test set protein pocket. QVina⁴⁵ computes all Vina scores, while RDKit computes the chemical properties.

The violin plots in Fig. 3 show the distribution of these metrics, capturing both the spread and central tendency of values across all protein pockets.

Fig. 3.

Comprehensive distribution analysis of generated ligands across various properties relevant to drug-likeness and binding.

• Figure 3(a) illustrates the QED (Quantitative Estimate of Drug-likeness) scores. There is a notable density between 0.2 and 0.7, suggesting the model frequently generates molecules with acceptable QED values.

• Figure 3(b) presents the Synthetic Accessibility (SA) scores. The majority of generated molecules fall within a synthetically feasible range, suggesting that LigGen tends to generate compounds that are not only drug-like but also potentially synthesizable.

• Figure 3(c) shows the distribution of LogP values, reflecting hydrophobicity. The majority of the data is concentrated between – 1 to + 5, with the median near 1.5 to 2, aligning well with the desirable LogP range for drug-like molecules (typically between 0 and 5).

• Figure 3(d) evaluates the extent to which the generated ligands comply with Lipinski’s Rule of Five. The majority of molecules satisfy all 5 rules, which suggests most generated ligands are highly drug-like.

• Figure 3(e) reports molecular diversity which is calculated as the average pairwise dissimilarity (1 - Tanimoto similarity) between all pairs of ligands generated for a given protein pocket. The distribution centers around 0.7, indicating moderate to high molecular diversity across most pockets.

• Figure 3(f) shows the distribution of binding affinities, as predicted by QVina, A median around -8 kcal/mol indicates that many generated compounds are predicted to bind strongly to their respective pockets.

Ligand efficiency and molecular obesity analysis

To address the potential bias of molecular obesity where binding affinity improves simply due to increased molecular size rather than specific interactions, we analyzed the Ligand Efficiency (LE) of the generated library. LE normalizes the binding affinity by the heavy atom count (), providing a robust metric for interaction quality.

Figure 4 presents the density distribution of LE against Molecular Weight (MW). As expected in medicinal chemistry, a negative correlation (Pearson r = – 0.78) is observed, reflecting the diminishing returns of adding atoms to a scaffold. However, crucial for de novo design, the distribution demonstrates that LigGen resists this decay effectively:

• Retention of efficiency: The global mean LE is 0.32, sitting comfortably above the widely accepted drug-like threshold of 0.3.

• High density zone: The highest density of generated ligands (indicated by the darkest blue hexbins) clusters in the efficient region () for molecular weights between 200 and 450 Da.

• Control of molecular obesity: The strong correlation between MW and Vina score is a known bias in docking functions. However, LigGen’s multi-objective function counteracts this by incorporating a Synthesizability (SA) penalty. Since the SA score generally correlates with molecular complexity and size, it acts as a regularizer. The algorithm is discouraged from simply adding atoms to improve the Vina score unless those atoms contribute specific, high-quality interactions that outweigh the penalty of increased synthetic complexity.

Fig. 4.

Distribution analysis of Ligand Efficiency (LE) versus Molecular Weight (MW) for 10,000 generated ligands. The hexbin density plot reveals that while a natural decline in efficiency is observed with increasing size (Pearson r = – 0.78), the majority of LigGen-generated compounds (dark blue regions) concentrate above the ’drug-like’ threshold of LE 0.3 (dashed black line). The global mean LE of 0.32 indicates that the generative process avoids ’molecular obesity’ by prioritizing high-quality interactions per atom.

Ligand–protein interaction analysis

The ligand-protein interaction profiles were analyzed using LigPlot+⁴⁶. To analyze the interaction quality in the most favorable binding environments identified by LigGen, we selected the three protein targets (PDB IDs: 3O96, 1H36, 1E8H) that exhibited the highest average binding affinities across the CrossDocked test set. For each of these targets, a representative generated ligand was selected for 2D interaction profiling. The interaction diagrams (Figure  5) reveal detailed hydrogen bonding and hydrophobic interactions between the ligands and key amino acid residues within the binding sites. In the protein pocket 3O96, the ligand forms strong hydrogen bonds with residues Thr82(A), Asn54(A), Glu17(A), and Arg273(A), with bond distances ranging from  2.89 Å to 3.19 Å, suggesting stable ligand accommodation in the pocket. Similarly, in the protein pocket 1H36, the ligand exhibits significant interactions with residues Tyr420(A), Tyr609(A), and adjacent polar groups, establishing hydrogen bonds between  2.82 Å and 3.24 Å, which indicates strong stabilization within the active site. For the protein pocket 1E8H, multiple hydrogen bonds were observed with residues Ser101(A), Ile102(A) Gly103(A), Gly110(A), Ser257(A), Phe254(A) and Lys545(A), showing distances between 2.89 Å and 3.28 Å, further supported by several surrounding hydrophobic contacts that enhance ligand stability. Across all three complexes, the combination of hydrogen bonding and hydrophobic interactions demonstrates efficient ligand accommodation within the binding pockets, implying a high binding affinity and potential inhibitory activity of the screened ligands.

Fig. 5.

2D ligand–protein interaction diagrams generated using LigPlot+. The three proteins (3O96, 1H36, 1E8H) were selected as they yielded the highest average Vina scores in the test set. A representative ligand from each target is shown to highlight the formation of specific hydrogen bonds (green dotted lines) and hydrophobic contacts (red arcs).

Comparative analysis of LigGen for alzheimer’s-related targets: BACE1, MAO-B, and AChE

While section “Crossdocked dataset” established the statistical performance of LigGen on the standardized CrossDocked benchmark, this section focuses on three specific, clinically relevant targets: Beta-secretase 1 (BACE1), Monoamine Oxidase B (MAO-B), and Acetylcholinesterase (AChE). For these high-value targets, we shift our analysis from broad statistical metrics to specific biophysical assessments. Consequently, rather than relying on static 2D interaction plots (as in section “Ligand–protein interaction analysis”), we employ Molecular Dynamics (MD) simulations and MM/PBSA calculations (detailed in section “Molecular dynamics (MD) simulation and MM/PBSA binding energy analysis”) to provide a rigorous, dynamic analysis of the ligand-protein interactions. LigBuilder V3, Pocket2Mol, and REINVENT 4⁴⁷ are representative tools for diverse de novo design strategies: Genetic Algorithms, Deep Learning (Graph-based), and Reinforcement Learning (RL), respectively. Specifically, REINVENT was included to compare our fragment-based growth strategy against an established RL approach that optimizes whole-molecule generation using the same AutoDock Vina scoring function. For the REINVENT baseline, we utilized the REINVENT 4 framework configured with a ’staged learning’ strategy. The optimization was conducted over 20 steps per stage with a batch size of 128 sequences. The agent was rewarded using a geometric mean of three components: the AutoDock Vina score (weight = 0.7), QED (weight = 0.1), and a synthesizability filter based on custom SMARTS alerts (weight=0.2), ensuring a comparable multi-objective optimization to LigGen. We have generated results for three different targets with PDB IDs 2G94⁴⁸, 2V5Z⁴⁹, and 7D9O⁵⁰. The designed ligands were evaluated based on their Binding Affinities (using AutoDock Vina), Synthesizability score (estimates how easy it would be to synthesize a molecule) and Similarity score (how similar a molecule is with already available ligand databases).

A total of 100 ligands were generated for each protein using all three programs, LigGen, Pocket2Mol and LigBuilder. Subsequently, these compounds underwent docking using AutoDock-Vina.

Protein descriptions

Beta-secretase 1 (BACE1)

BACE1 is the major beta-secretase responsible for generating amyloid- peptides in neurons⁵¹. PDB with ID 2G94 is chosen for the structure of BACE1.

Monoamine Oxidase B (MAO B)

Monoamine oxidase B is a flavoenzyme found in the outer mitochondrial membrane, playing a role in the breakdown of monoamines⁵². PDB with ID 2V5Z is chosen for the structure of MAO-B.

Acetylcholinesterase (AChE)

Acetylcholinesterase is the enzyme that degrades acetylcholine, and inhibiting this enzyme can enhance cholinergic activity⁵³. PDB with ID 7D9O is chosen for the structure of AChE.

The structure of known top Inhibitors was extracted from the PDB files of their respective proteins and their structure is shown in Fig. 6. In fact, the inhibitor safinamide is a known drug for MAO-B, which is a therapeutic target for Parkinson’s disease.

Fig. 6.

This figure depicts the 3d structure of known Inhibitors with PDB ID ZPQ, SAG and HOL of targets 2G94, 2V5Z and 7D9O respectively. (a) Ligand ZPQ. (b) Ligand SAG. (c) Ligand HOL.

Docking score analysis

The docking score analysis evaluates the binding affinities of ligands generated by LigGen, LigBuilder V3, Pocket2Mol and Known Inhibitors using Autodock-Vina. This analysis includes average docking scores and the top docking scores for specific protein targets, providing insights into the effectiveness of different ligand generation methods in producing high-affinity ligands. Table 2 denotes the average docking of the ligands which were successfully docked using AutoDock-Vina. Tables 3,  4 and 5 lists the Autodock-Vina docking scores of the top 5 ligands with the least docking scores generated by LigGen, Pocket2Mol, LigBuilder, and known inhibitors for the target protein 2G94, 2V5Z, 7D9O respectively. The distributions of docking scores computed using AutoDock-Vina for ligands generated using LigGen, Pocket2Mol, LigBuilder V3 and REINVENT4 are shown in Fig.  7.

Table 2.

Average docking score computed using AutoDock-Vina.

Proteins	2G94	2V5Z	7D9O
LigGen	– 9.24	– 9.44	– 11.42
Pocket2Mol	– 7.02	– 8.31	– 10.20
LigBuilder	– 7.33	– 9.01	– 9.44
REINVENT	– 8.07	– 8.05	– 9.93

Table 3.

Autodock-vina Docking score of top 5 generated ligands for target 2G94.

S.No.	LigGen	P2M	LigB	REINV	Known
1	– 11.92	– 8.58	– 8.62	– 10.50	– 8.80
2	– 11.50	– 8.40	– 8.57	– 10.30
3	– 11.46	– 8.39	– 8.56	– 9.90
4	– 11.15	– 8.32	– 8.38	– 9.60
5	– 11.14	– 8.31	– 8.36	– 9.50

Abbreviations: P2M (Pocket2Mol), LigB (LigBuilder V3), REINV (REINVENT 4), Known (Native Inhibitor).

Table 4.

Autodock-vina Docking score of top 5 generated ligands for target 2V5Z. Abbreviations: P2M (Pocket2Mol), LigB (LigBuilder V3), REINV (REINVENT 4), Known (Native Inhibitor).

S.No.	LigGen	P2M	LigB	REINV	Known
1	– 13.50	– 12.82	– 11.51	– 11.60	– 10.32
2	– 13.14	– 11.49	– 11.47	– 11.40
3	– 12.81	– 11.22	– 11.46	– 11.20
4	– 12.60	– 11.22	– 11.46	– 11.10
5	– 12.33	– 11.07	– 11.45	– 11.10

Table 5.

Autodock-vina Docking score of top 5 generated ligands for target 7D9O. Abbreviations: P2M (Pocket2Mol), LigB (LigBuilder V3), REINV (REINVENT 4), Known (Native Inhibitor).

S.No.	LigGen	P2M	LigB	REINV	Known
1	– 15.45	– 14.28	– 10.54	– 13.40	– 12.24
2	– 15.37	– 14.19	– 10.53	– 13.40
3	– 14.22	– 14.12	– 10.53	– 13.10
4	– 14.16	– 13.78	– 10.53	– 12.90
5	– 13.97	– 13.51	– 10.52	– 12.80

Fig. 7.

Distribution of Docking Scores for Ligands Generated by LigGen, Pocket2Mol, LigBuilder and REINVENT for Proteins 2G94, 2V5Z, and 7D9O.

Structural comparison with native inhibitors

To validate that LigGen captures biologically relevant binding modes, we compared the predicted poses of our top-scoring generated ligands with the crystallographic conformations of the known native inhibitors (ZPQ for BACE1, SAG for MAO-B, and H0L for AChE).

As shown in Fig. 8, the generated ligands exhibit a high degree of spatial overlap with the native inhibitors. Quantitatively, the Tanimoto similarity (based on ECFP4 fingerprint)  between the top generated ligands and the native inhibitors was found to be 0.42 (2G94), 0.19 (2V5Z), and 0.45 (7D9O). These moderate scores indicate that LigGen is not simply reproducing the native scaffold (which would yield a score near 1.0) but is instead designing novel chemotypes that satisfy the same pharmacophoric requirements.

Fig. 8.

Structural superposition of the top LigGen-generated ligand (Cyan) and the native co-crystallized inhibitor (Magenta) within the binding pocket. (a) BACE1 (2G94). (b) MAO-B (2V5Z). (c) AChE (7D9O).

Synthesizability analysis

Synthesizability score of ligands was calculated using RDKit sascorer package. Sascorer package can characterize molecule synthetic accessibility as a score between 1 (easy to make) and 10 (very difficult to make). The count of ligands having synthesizability score smaller than the value of 5 is listed in Table 6 and Fig. 9 depicts the distribution of synthesizability score for ligands generated for different targets. It is observed that for certain targets, LigGen produces a distribution with slightly higher average SA scores compared to baseline methods. This reflects the algorithm’s prioritization of high-affinity binding modes over trivial synthesizability. While some baseline methods optimize for simple, drug-like heuristics at the cost of binding potency, LigGen effectively explores the “complex-but-synthesizable” chemical space to identify deeper energy minima within the binding pocket.

Table 6.

Count of ligands having Synthesizability score smaller than the value of 5.

Proteins	2G94	2V5Z	7D9O
LigGen	82	96	89
Pocket2Mol	99	100	97
LigBuilder	100	100	100
REINVENT	100	99	100

Fig. 9.

Distribution of Synthesizability (SA) Scores for ligands generated for targets 2G94, 2V5Z, and 7D9O. The plots illustrate the full range of SA scores for all 100 generated ligands per tool, where lower values indicate better synthesizability.

Ligand similarity comparison

In this section, we assess the similarity of the generated ligands to those available in ChEMBL library using SwissSimilarity. By comparing our ligands with the extensive collection in the ChEMBL database, we aim to evaluate their novelty. This comparison helps in understanding how closely our ligands resemble existing compounds and provides insights into their uniqueness and potential for further research or development. To assess the novelty of the generated compounds, we performed a similarity screening against the ChEMBL database (version 29) using the SwissSimilarity web server. We utilized Extended Connectivity Fingerprints (ECFP4) for this analysis, as they robustly capture topological substructures and are widely accepted for structural similarity comparisons. The similarity score ranges from 0 (completely different) to 1 (identical). As illustrated in Fig. 10, the distribution of similarity scores for LigGen-derived ligands centers around 0.3–0.4 across all three targets. This low average similarity indicates that LigGen is not merely retrieving known inhibitors from the chemical space but is successfully generating novel chemical entities with distinct scaffolds. This ability to access unexplored regions of chemical space while maintaining high binding affinity is a key advantage of the de novo design approach. To further validate this, we identified the closest ChEMBL matches for the top-docking ligand of each target:

Fig. 10.

Distribution of similarity score ligands for different targets.

• BACE1 (2G94): The top ligand showed a low maximum similarity of 0.276 (CHEMBL3431601), indicating a highly novel scaffold.

• MAO-B (2V5Z): The closest match had a similarity of 0.363 (CHEMBL4545516), suggesting a distinct chemotype despite the restrictive binding pocket.

• AChE (7D9O): The top ligand exhibited a moderate similarity of 0.50 (CHEMBL3329368), implying that while it shares certain pharmacophoric features with known actives, it retains significant structural differentiation.

This ability to access unexplored regions of chemical space (similarity ) while maintaining high binding affinity is a key advantage of the de novo design approach.

Molecular dynamics (MD) simulation and MM/PBSA binding energy analysis

To further evaluate the stability and binding efficiency of the top-performing ligands generated by LigGen, Pocket2Mol, and LigBuilder, Molecular Dynamics (MD) simulations followed by MM/PBSA⁵⁴ free energy calculations were performed. The ligand selected for each tool-protein pair corresponded to the compound with the best (lowest) docking score from the AutoDock Vina results.

MD simulation protocol

MD simulations were carried out for 100 ns for each protein-ligand complex using GROMACS⁵⁵. In total, 9 simulations were conducted (3 tools × 3 protein targets). Each trajectory was saved every 10 ps, resulting in 10,000 frames per simulation.

• Systems simulated: 2G94 (BACE1), 2V5Z (MAO-B), 7D9O (AChE) with best-scoring ligand from each tool.

• Total simulation time: 100 ns per system.

• Force field: AMBER99SB⁵⁶ for protein, GAFF⁵⁷ for ligands.

• Solvation model: SPC/E water model in a dodecahedral box with Na⁺/Cl^– ions added for system neutralization.

• Equilibration: Minimization, NVT, and NPT steps before production run.

MM/PBSA free energy calculation

The MM/PBSA (Molecular Mechanics Poisson-Boltzmann Surface Area) method was used to estimate the binding free energies of the complexes over the last 20 ns of the MD trajectory (i.e., frames 8000–10000) using a step size of 20, resulting in 100 frames per simulation.

The average MM/PBSA binding energy across these frames is summarized in Table 7. A lower (more negative) value corresponds to stronger predicted binding affinity.

Table 7.

Average MM/PBSA binding free energy (kcal/mol) for top ligands from each tool.

Proteins	2G94	2V5Z	7D9O
LigGen	– 20.47	– 34.89	– 28.55
Pocket2Mol	– 15.18	– 31.50	– 20.38
LigBuilder	– 3.71	– 29.45	– 20.14

As shown in the table, LigGen consistently achieved the most favorable MM/PBSA scores across all targets, reinforcing its superior performance seen earlier in docking results. The performance gap is especially significant for BACE1 (PDB ID: 2G94) and AChE (PDB ID: 7D9O), where LigGen’s ligand displayed a much stronger binding energy compared to the other tools. The distribution of MM/PBSA scores over the sampled 100 frames (Supplementary Fig. S1) provides insight into the binding-energy fluctuations during the final phase of the simulation. To assess the structural stability of the protein–ligand complexes over the 100 ns MD simulations, we analyzed the Root Mean Square Deviation (RMSD) and Radius of Gyration (Rg) for all systems. The RMSD plots indicate that all complexes reached equilibrium early and remained stable throughout the simulation, with the exception of the ligand generated by LigBuilder for BACE1 (PDB ID: 2G94), which showed signs of instability. The Rg plots show consistent structural compactness across systems, further confirming stability. These analyses are presented in Supplementary Figs. S2 and S3.

Conclusion

In this study, we developed a novel de-novo ligand generation tool which used fragments as building blocks. The fragment generation itself is based on generative AI based on RNN which was pretrained on ChEMBL dataset and fine-tuned on LigBuilder’s fragment library. We evaluated LigGen’s performance on CrossDock dataset and three Alzheimer’s-related proteins to provide a comparison with several state-of-the-art de novo drug design tools, including Pocket2Mol, LigBuilder V3, and others. On the CrossDock dataset, LigGen demonstrated competitive binding affinities, with a Vina score of -7.508 ± 1.83, outperforming all baseline methods like 3D-SBDD, FLAG, and DrugGPS. LigGen also showcased strong diversity in the generated molecules while maintaining reasonable synthesizability. Although some tools like Pocket2Mol showed higher QED values, LigGen consistently produced ligands with good Lipinski compliance, suggesting its balance between drug-likeness and innovation.

In the Alzheimer’s-related protein analysis, ligands produced by LigGen consistently showed better binding affinities compared to those from LigBuilder and Pocket2Mol. LigGen demonstrated a balanced approach between ligand synthesizability and structural complexity, as seen in its ability to produce ligands with moderate synthesizability scores.

To facilitate broader adoption and usability, we have also developed a free-to-use web server for ligand generation, accessible at⁵⁸. This allows researchers to generate ligands conveniently using LigGen without requiring local installations.

Overall, the results indicate that LigGen is a robust tool for fragment-based ligand generation, capable of producing high-quality ligands with competitive docking scores and favourable synthesizability. LigGen provides a novel method for generating ligands using a fragment-based de novo drug design approach. While the optimization process relies on AutoDock Vina, introducing a potential bias towards this scoring function, the robustness of the generated ligands was orthogonally validated using 100ns Molecular Dynamics simulations. Furthermore, this explicit optimization distinguishes LigGen from data-driven deep learning baselines (e.g., Pocket2Mol) which exhibit “implicit circularity.” Since these models are trained on curated datasets of protein-ligand complexes already selected for high affinity, their generative capabilities are inherently dependent on the specific distributions of known binders. In contrast, LigGen’s physics-driven approach minimizes energy independently of training set biases, allowing for the discovery of novel binding modes that may not be represented in existing structural databases.

Author contributions

 A.Y : Data curation, formal analysis, investigation, methodology, LigGen software development, visualization, validation, writing – original draft, writing – review and editing. N.A.M: Conceptualization, investigation, project administration, supervision, funding acquisition, writing – review and editing.

Funding

This work was supported by the Science and Engineering Research Board (SERB), Government of India, under the SERB-SURE scheme (Grant No. SUR/2022/005296).

Data availability

The LigGen tool, processed results, and the complete computational pipeline used to generate the findings reported in this study are available on Zenodo at the following link⁵⁹. The source code and implementation details for LigGen are freely available on GitHub at https://github.com/neurocare-iiitd/LigGen.

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-48239-2.