ScrambleBench: a workflow for comparative assessment of structure-based de novo generative models

Abstract

Generative artificial intelligence (AI) has rapidly advanced over the past decade in the field of drug discovery, particularly for the de novo design of small molecules based on target protein structures. While generative AI has the potential to complement traditional structure-based drug design (SBDD) to discover novel hits, their performance is often assessed using non-standardized evaluation criteria. As the number of generative AI models continue to grow, it becomes increasingly important to determine whether these tools are sufficiently robust and reliable for integration into medicinal chemistry workflow. Here, we propose ScrambleBench, a benchmarking workflow designed to evaluate structure-based generative AI models that align with medicinal chemists’ practical objectives to identify chemically diverse, drug-like hit candidates that adopt plausible binding conformations and exhibit favourable docking affinities. Using six representative models (Pocket2Mol, PocketFlow, Lingo3DMol, DiffSBDD, PMDM, and Chemistry42), we systematically assessed performance across diverse target classes, including two GPCRs, two kinases, and two hydrolases. While some generative models show superior performance for particular evaluation endpoints, none demonstrates overall dominance across all evaluated criteria. Notably, despite benchmarked proteins (e.g., CDK2, GSK3β) being present in the training datasets, the models still show limited generalization to target binding sites, which resulted in high redocking RMSD values and low virtual hit rates. Our results highlight the importance of evaluating chemical diversity explicitly and using the recently proposed metrics such as Hamiltonian Diversity (HamDiv) which assess both quantity and dissimilarity of a molecular set. Furthermore, as many generated ligands fail to meaningfully engage the target active site, we propose that future generative frameworks incorporate improved loss functions that place greater emphasis on drug-like physicochemical properties and correct pharmacophore recognition.

Scientific contribution

While numerous de novo generative models have been proposed for structure-based molecular design, objective comparison between methods remains challenging due to inconsistent benchmarking practices and heterogeneous evaluation criteria. ScrambleBench introduces a unified and reproducible benchmarking workflow that integrates diversity analysis, conformational validity assessment, docking reproducibility, pharmacophore matching, and virtual hit rate evaluation within a single framework. By systematically comparing representative generative models using common datasets and standardized assessment criteria, this work advances the field by enabling transparent evaluation model performance and practical applicability. Overall, ScrambleBench provides a holistic medicinal chemistry-oriented framework that identifies methodological strengths, limitations, and opportunities for future model development.

Graphical Abstract

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s13321-026-01254-x.

Introduction

Over the past decade, generative artificial intelligence (AI) models have emerged as a prominent approach for de novo ligand design targeting protein binding pockets. These models are commonly benchmarked based on the generated ligand’s geometric and energy validity to understand whether they can design structurally valid ligands with predicted binding affinity [1]. While many published studies report improvements over previous state-of-the-art (SOTA) models, benchmarking practices often rely on selectively chosen metrics that favour the proposed method, making it challenging to assess true model performance in an un-biased manner. Indeed, multiple studies have reported issues such as limited number of ligands being generated, steric clashes, ligand diversity, and unrealistic ligand conformations, which are often inconsistent with medicinal chemists’ intuition and practical requirements [2–4]. Furthermore, some models evaluate performance using datasets such as Directory of Useful Decoys, Enhanced (DUD-E) [5] or specific targets to evaluate other existing generative models, thus convoluting whether they can generate valid ligands for unseen proteins [3, 6]. As a result, a lack of standardized and systematic benchmarking makes it difficult for medicinal chemists to evaluate a model’s practical utility until they attempt to reproduce and deploy it on their respective protein targets. Here, we provide streamlined and systematic benchmarking analysis for several widely used generative AI models, with the goal of identifying common limitations and guiding future methodological improvements. Specifically, we benchmarked three autoregressive models, two diffusion models, and one hybrid model (Table 1) across six clinically relevant protein targets using commonly adopted evaluation metrics. Among these models, “Chemistry42” [7] represents an industry-scale generative platform that has been reported to support multiple drug discovery programs, including compounds that have advanced to phase 2a clinical trial [8, 9], offering a practical reference point alongside academic methods.

Table 1.

Examples of published pocket aware generative model and its important information detailing architecture and main limitation outside of their inherent model type

Model	Model type	Main novel architecture	Limitation
Pocket2Mol [11]	Autoregressive	Utilise geometric vector perceptron (GVP) to represent 3D atoms	Unrealistic ligand topology [4, 6, 10]
Lingo3DMol [6]	Autoregressive + Large Language Model (LLM)	Build molecule fragment-wise towards predicted non-covalent interaction (NCI) residues	Low diversity from first depth search
PocketFlow [19]	Autoregressive + Flow model	Using upgraded geometric double bottleneck perceptron (GDBP) to accurately learn training dataset distribution, and modifying un-drug like molecule through domain knowledge	No explicit training of binding affinity
DiffSBDD [10]	Diffusion model	First applied diffusion model inspired by Hoogeboom [12]	Poor representation to train bond length [2]
PMDM [2]	Diffusion model	Dual diffusion model to include bond length and improve intrinsic topology structure	Involve Open Babel to infer bond type during final generation step
Chemistry42 [7]	Mixed model	Combine multiple generative models with reward function to diversify ligand structure	Proprietary and commercial licensing

To date, there are two popular deep learning architectures used for pocket-based generative models: autoregressive and diffusion-based models [1]. Autoregressive model builds atoms sequentially, and predicts atom positions, atom types, and bond types at each step based on the partially constructed structure [6]. In contrast, diffusion-based models typically initialize a pre-determined number of atoms near the pocket center and iteratively update atomic (and sometimes bond) information on every diffusion step [2]. As both models use protein pockets as a restraining context, they are required to learn rotational and translational equivariance of the protein–ligand complex, commonly through E(3) or SE(3) Equivariant Graph Neural Network (EGNN) architecture. Despite their conceptual simplicity, autoregressive models have faced substantial criticism due to their susceptibility to generating unrealistic or chemically unstable scaffolds, motivating a recent shift toward diffusion-based models [6, 10] (Table 1).

In parallel with model development, several benchmarking methods have been developed to evaluate these generative models beyond the metrics reported in their original publications. Early efforts include the first pocket-based generative model, Pocket2Mol, which employed Vina scores and Kullback–Leibler (KL) divergence as three-dimensional (3D) related metrics to demonstrate its state-of-the-art understanding of the pocket space [11]. While these metrics represented important initial attempts, they failed to capture the full extent of structural realism. Subsequent analyses revealed that Pocket2Mol frequently generates ligands with unrealistic topologies such as puckered aromatic rings and incorrect bond lengths [4, 10]. To address these shortcomings, geometry focused evaluation tools such as PoseBusters, which assesses intermolecular and intramolecular distance geometry [14], and PoseCheck, which evaluates Vina redocking RMSD, strain energy, van der Waal (vdW) distance, and protein–ligand interaction fingerprint (PLIF) [15], were introduced. While some of these tools provide important insights into ligand geometry and interaction quality, they are often applied within workflows that evaluate redocked poses of de novo ligands rather than the originally generated conformations. As a result, such evaluations may not fully capture whether generative models can directly produce physically realistic ligand conformations [13].

To overcome this limitation, DrugPose [4] and GenBench3D [13] were developed to directly evaluate the generated conformations. DrugPose checks whether the generated pose can be reproduced by smina docking, while GenBench3D calculates the likelihood of geometric patterns (bond angle, bond length, and torsion angle) without performing docking [13]. More recently, HEAD-TED was proposed as an AI based model to identify high-energy atoms based on their local atomic environments (HEAD) and to detect unfavourable torsions using an energy threshold (TED) model [1]. This approach is conceptually similar to GenBench3D in its objective to detect abnormal geometric pattern. While these methods provide important advances in evaluating geometric validity and docking consistency, they often assess individual aspects of generalised ligands in isolation.

An additional challenge in benchmarking structure-based generative models lies in the selection of appropriate test sets comprising proteins unseen during training. However, test set choices vary substantially across studies. Some models adopt test set from the CrossDocked2020 dataset [2, 10, 16], Others use alternative benchmarks (e.g., DUD-E) [6], while several studies rely on curated clinical protein targets as case studies [3, 17]. Although CrossDocked2020 [18] and DUD-E encompass major protein families (GPCRs, kinases, and proteases), there is a limited transparency regarding whether the generative models preferentially perform better on specific protein classes. In this context, case studies aiming for addressing these unanswered questions can provide valuable insights into real-world applicability. However, many studies restrict their evaluation to one or two protein targets [2, 19, 20], with a strong bias toward kinases and proteases, particularly SARS-CoV-2 3CL protease (Supplementary Fig. 1).

Here, we introduce ScrambleBench an integrative framework that address these limitations by systematically benchmarking representative structure-based generative models across multiple protein classes using a unified evaluation framework, enabling direct comparison of model performance, generalizability, and practical hit-generation capability from a medicinal chemistry perspective. This integrative approach enables a more comprehensive assessment of model performance compared to existing benchmarks that focus on isolated evaluation criteria.

Methodology

Protein and pocket pre-processing

The structures of six protein targets were obtained from the RCSB Protein Data Bank (PDB) (Table 2). Protein preparation and downstream analysis were performed using Schrödinger Maestro version 2025.4 (v14.6.125). Protein–ligand complexes were prepared using the Protein Preparation Workflow [21], including protonation at pH 7.4 ± 1.0 using the OPLS4 forcefield [22], conversion of selenomethionine to methionine, correction of missing chains, and removal of crystallographic water molecules and ions.

Table 2.

PDB information of the targets used for benchmarking

PDB ID	Protein	Ligand	Resolution (Å)
6BQH	5-Hydroxytryptamine 2C receptor (5-HT2C)	Ritanserin	2.70
6LUQ	Dopamine receptor D2 (DRD2)	Haloperidol	3.10
8H6T	Cyclin-dependent kinase 2 (CDK2)	‘Molecule 13’	3.00
4ACD	Glycogen synthase kinase 3 beta (GSK3β)	‘Molecule 18’	2.60
7JSU	SARS-CoV-2 3C-like protease (3CLpro)	GC376	1.83
3CL0	H5N1 neuraminidase H274Y (NA)	Oseltamivir	2.20

For generative models requiring pocket-only inputs rather than full protein–ligand complexes, pocket structures were prepared using the PMDM GitHub script [2]. Briefly, residues surrounding the bound ligand were extracted using the Open Drug Discovery Toolkit (ODDT) [23]. The extracted residues were then deprotonated using Open Babel [24]. A protein–ligand distance cutoff of 10 Å was applied, consistent with the cut-off commonly used in model training datasets [11].

De novo ligand generation

To determine each model’s ability to generate diverse virtual hit candidates, we prompted each model to generate 100, 200, and 500 ligands using the “num_sample” parameter. The centroids of the protein binding pockets were computed using RDKit [25] based on the native ligand coordinates and used to define the space for molecular generation. Unless otherwise specified, default parameters were retained for each model. All models were obtained from their respective GitHub repositories trained on the CrossDocked2020 Dataset [18] under full-atom conditioning.

For Pocket2Mol, a bounding box size (“bbox_size”) of 16 Å was used for all protein targets, except H5N1 neuraminidase (NA), where ligand generation was insufficient and the bounding box was increased to 18 Å. To mitigate the generation of fragmented ligands, the default “beam_size” parameter was increased to 50 with 100 iterations. If the desired number of ligands (“num_sample”) was not achieved, the “beam_size” was further increased to 100 and 150 as required. For PMDM, a “batch_size” of 5 was used to avoid out-of-memory (OOM) errors during ligand generation. For Lingo3DMol, the maximum runtime was increased from one hour to three hours to accommodate larger “num_sample” size. All de novo ligand generations were carried out using 1 node of Quadro RTX 6000 with max memory of 23,040 MiB.

Generated ligand selection

In cases where models produced more ligands than the specified “num_sample”, ligands were subsampled in a model-dependant manner. For Lingo3DMol and Pocket2Mol, the last “num_sample” ligands generated were retained. As Chemistry42 generated thousands of ligands per run, ligands were randomly sampled instead. Ligands that passed RDKit validity checks were assigned unique ID numbers and carried forward for further analysis.

Generated ligand analysis

Generated ligands were evaluated using the following categories of metrics:

(1) molecular properties including molecular weight (MW), lipophilicity (log P), synthetic accessibility score (SA Score), and quantitative estimate of drug-likeness (QED) score were calculated using RDKit; (2) molecular performance measures the ligand uniqueness, diversity measured using Hamiltonian diversity (HamDiv)—a diversity metrics that considers both quantity and dissimilarity between ligands [26]—and the number of unique generic Bermis-Murcko (BM) scaffolds; (3) geometric validation estimates the Validity2D and Validity3D scores defined by GenBench3D; and (4) binding properties predicts the binding affinity using Autodock Vina [27] and Schrödinger Glide SP [28], redocking root mean square deviation (RMSD), and pharmacophore matching score calculated using Schrödinger Phase [29].

Diversity metrics

Unlike other metrics used for assessing molecular diversity, the Hamiltonian Diversity (HamDiv) considers both quantity of the ligands and the degree of similarity among them within a library or dataset [26]. Libraries dominated by highly similar compounds therefore score poorly compared with those containing fewer but more diverse ligands. This metric is well suited for our study, as it enables benchmarking of each model’s ability to generate a large set of structurally diverse candidates and to assess potential overfitting during training. HamDiv was calculated using the 2-opt search algorithm [26] with two different distance metrics: Maximum Common Edge Subgraph (MCES) and Extended Connectivity Fingerprint (ECFP) Tanimoto distance. Although ECFP Tanimoto distance, based on Morgan Fingerprints (radius = 2 and fpSize = 2048), is widely used, it primarily captures local atomic environments and may not fully reflect global scaffold similarity between molecules [30]. Hence, similar to other fingerprint-based similarity, this metric can behave differently depending on the size of the molecules [31]. Therefore, MCES distance was employed to better quantify shared substructures and overall topological similarity between ligand pairs. MCES calculations were performed using the RDKit’s RApid Similarity CALculation (RASCAL) algorithm [32] allowing partial atomic ring matches (completeAromaticRings = False), up to 2000 bond matches, and treating halogens [Br, Cl, F, and I] as equivalent atoms, as well as atoms [S, O]. While inaccurate MCES can influence the result for Tanimoto Distance via triangle inequality [26], we found that using a more flexible parameter of timeout of 1000s and initial similarity threshold of 0.5 did not affect our overall conclusion of this study, balancing accuracy and computational efficiency (Supplementary Fig. 2).

The Tanimoto distance (Td(MCES)) was calculated using the following equation:

$$Td\left(M,C,E,S\right)=\frac{E\left(M,C,E,S\right)+V(MCES)}{E\left(M,o,l,1\right)+V\left(M,o,l,1\right)+E\left(M,o,l,2\right)+V\left(M,o,l,2\right)-[E\left(M,C,E,S\right)+V\left(M,C,E,S\right)]}$$

where E(MCES) and V(MCES) represent the edges (bond) and vertices (atom) of the matched MCES substructure of first and second molecule, respectively. As the RASCAL algorithm considers both atom and bond, both parts were included into the equation.

Unique number of Generic BM Scaffolds were computed for each ligand and subsequently generalized by converting all atoms to carbon and all bonds to single bonds. In cases where atoms capable of forming more than four bonds (e.g., sulphur) resulted in unsanitizable structures, the scaffolds were retained without sanitization, allowing carbon atoms to exceed typical valence constraints.

Geometric validation metric

To validate generated ligand conformation, GenBench3D benchmarking model was used to score Validity3D to assess each ligand’s bond length, bond angle, and its planarity. Validity3D is defined as the proportion of ligands that have valid conformations over total number of ligands evaluated [13].

The GenBench3D model was fine-tuned to accept ligands prepared by Schrödinger's LigPrep module [33]. Both undocked and docked ligand conformations were used as training data to capture valid geometries under different conditions. Approximately 4 million SMILES from the Enamine Screening Collection were prepared at pH 7.4 ± 1.0, yielding 10,035,333 ligands. However, due to computational constraint, a single protonation state or isomer was randomly selected per molecule, resulting in 4,273,609 undocked ligands. These 10 million prepared ligands were docked against AKT serine/threonine kinase 1 (AKT1) (PDB ID: 7NH5) using the Glide virtual screening workflow. The top 10% from high throughput virtual screening (HTVS) were subjected to Glide SP docking, and the top 5% were retained, yielding 45,652 docked ligands.

The bandwidth parameters controlling the smoothness of bond length and bond angle distributions were optimized independently until > 99% of both docked and undocked ligands were classified as valid. Final bandwidth values of 0.03 for bond length and 0.40 for bond angle were chosen for downstream analysis (Supplementary Fig. 3).

Binding property metrics

Ligand binding was evaluated using the “in-place” and “min-in-place” docking scores from both AutoDock Vina and Glide SP as implemented in GenBench3D. MMFF94 forcefield minimisation [34] was applied to resolve steric clashes with the protein. Models that correctly capture the context of protein pocket should generate ligands with favourable docking scores without requiring conformational resampling. The GenBench3D tool was modified to accept proteins prepared directly via the Schrödinger Protein Preparation Workflow. The tool was also modified to exclude fragmented or invalid ligands after MMFF94 minimisation.

To further assess pose stability, the generated ligands were redocked using Vina (via EasyDock tool [35]) and Glide SP. For Vina redocking, an exhaustiveness of 32 was employed for consistent docking results, generating five poses for each ligand. For Glide SP redocking, OPLS_2005 [36] was used as forcefield, planarity was forced and intramolecular hydrogen bonds were rewarded. Specifically for kinases, planarity was not forced and 20 poses were generated instead as default settings failed to reproduce known cross-docking pose of maleimide-based inhibitors (Supplementary Fig. 4).

RMSD calculations were performed by converting final Vina pdbqt files into RDKit Mol object using Meeko’s RDKitMolCreate.from_pdbqt_mol function [37]. Redocked ligands were neutralised and stripped of most hydrogens for cases involving tautomeric differences, MCES based atom mapping was applied using RDKit RASCAL algorithm. The RMSD between the “in-place” conformation and redocked conformation was then calculated using RDKit 2025.9 CalcRMS function, with bond order agonistic matching applied as a fallback when necessary.

Pharmacophore scoring

Another metric to validate generated ligand conformation besides RMSD and docking score was to use pharmacophore score. Pharmacophore hypotheses were manually constructed using Schrödinger Phase based on known ligand–protein interaction patterns from the literature (Table 3, Supplementary Fig. 5). For GPCRs, conserved aspartate interactions and selectivity-determining residues were included [38, 39]. For kinases, hinge region interactions were selected. The pharmacophore for SARS-CoV-2 3CL protease was selected based on the common interactions between the covalent ligand GC376 and non-covalent ligand WU-04 (PDB ID: 7EN8) [40], while the neuraminidase pharmacophore was selected based on its non-covalent interactions. Excluded volume constraints were generated automatically. PhaseScreenScore was computed for each ligand to quantify pharmacophore compatibility.

Table 3.

Pharmacophore hypothesis information generated from schrödinger phase

Protein target	Pharmacophore#	Residue interaction	Distance / Å	Matched pharmacophore	Other constraints
5-HT2C	P	D134	3	2/2
	R/H	W324	3
DRD2	R/H	F110	3	3/3
	R/H	W407	4
	P	D114	3
CDK2	R/H	I10	3	3/5	Both donor pharmacophores are grouped. Must match at least ½
	H/R	F80	3
	D/P	L83	2
	A/N	L83	2
	D/P	E81	2
GSK3β	A/N	K85	2	4/5
	R/H	V70	3
	A/N	V135	2
	R/H	V135	2
	R/H	I62	3
3CLpro	H/R	M49	2	3/5
	A/N	E166	2
	D/P	E166	2
	A/N	H163	2
	A/N	H164	2
NA	P/D	E119	2.5	3/3
	N/A	R118	2
	A/N	R152	2

^#For pharmacophore, P = positive, N = negative, A = hydrogen bond acceptor, D = hydrogen bond donor, H = hydrophobic, R = ring

Virtual hit rate calculation and best ligand selection

Although many published studies report favourable distributions of drug-like properties, such summaries do not reflect the proportion of ligands simultaneously satisfying multiple criteria. Hence, it is essential to filter ligands with multiple properties and evaluate them in holistic approach. Although some studies have been carried out using their best molecule, one molecule does not represent the properties of the whole generated ligands. To address this, we define the virtual hit rate as the fraction of generated ligands meeting a predefined set of holistic filters, providing a more representative measure of model performance.

To calculate the virtual hit rate, the ligands were sequentially filtered based on: (i) The redocked ligand from Glide SP need to match the pharmacophore hypothesis generated from Schrödinger Phase without conformational resampling. (ii) Glide SP docking score of <  − 8 kcal/mol, and (iii) QED > 0.5. Glide SP was chosen over Vina for virtual hit criteria, as Glide SP penalized intramolecular ligand strain [13] for invalid molecules. If no ligand passed a given filter, the best performing ligand according to that metric was retained.

The best generated ligand was selected from the filtered pool on combined consideration of chemical structure, docking score, QED, PhaseScreenScore, and RMSD between “in-place” and redocked conformations.

Results

Generative models can generate more than 100 diverse ligands

We first assessed whether each generative model could produce numerous diverse ligands across all six protein targets. All models except Pocket2Mol and Lingo3DMol can generate up to 500 diverse ligands for every target (Fig. 1). Among them, diffusion-based models performed particularly well in terms of diversity. DiffSBDD achieved the highest Hamiltonian diversity (HamDiv) based on the MCES metric, while PMDM generated the largest number of unique generic Bemis-Murcko (BM) scaffolds, reaching 500 unique scaffolds for five of six targets (Fig. 1B). These results indicate that PMDM excels at scaffold-level diversity, whereas DiffSBDD generated scaffolds with least Maximum Common Substructure (MCS) across targets.

Fig. 1.

Diffusion models generated the most diverse ligands for most tested protein targets. Line plot of the “num_sample” of generated ligand of each generative model against the A HamDiv MCES Tanimoto Distance and B the number of unique BM scaffold. The “num_sample” for control group represents the number of known active ligands for specified protein targets collected from the IUPHAR database. Error bands for each line correspond to 3 × standard error (SE) using triplicates

Other models such as PocketFlow and Chemistry42, exhibited moderate HamDiv MCES and number of generic BM scaffold score, occasionally outperforming diffusion-based models for hydrolase targets (Fig. 1B). In contrast, Pocket2Mol and Lingo3DMol consistently generated fewer unique ligands for several proteins and scored lowest across both diversity metrics.

Comparison between HamDiv MCES and HamDiv ECFP revealed that both metrics could distinguish autoregressive models from the diffusion-based models. However, HamDiv ECFP suggested that PMDM and DiffSBDD have comparable diversity across all sample size (“num_sample”) (Supplementary Fig. 6). Moreover, HamDiv ECFP ranked PocketFlow as more diverse than Chemistry42, whereas both unique generic BM scaffold counts and HamDiv MCES indicated the opposite trend. These discrepancies highlight the limitations of fingerprint-based diversity metrics alone.

Our results also showed that both number of unique Generic BM Scaffolds and HamDiv MCES can be used to distinguish ligand diversity compared to existing method of Average Pairwise Tanimoto Distance using ECFP fingerprint (Supplementary Fig. 7). While Average Pairwise ECFP Tanimoto Distance supported that Pocket2Mol and Lingo3DMol relatively have lower diversity than other models, both models have a decent score above 0.6 for all protein targets, suggesting comparable diversity compared to diffusion-based models.

Generative models analysis on GPCRs

As we have shown that most models could generate more than 100 ligands, we had used the dataset of generated ligands with “num_sample” of 500 for subsequent analyses. All models generated unique and RDKit-valid compounds for both 5-HT_2C (Table 4) and DRD2 (Table 5), although Lingo3DMol was limited to 200 ligands. PMDM required the longest time, consuming approximately 200 min and 300 min for 5-HT_2C (Table 4) and DRD2 (Table 5), respectively.

Table 4.

Total ligands generated, Validity2D, and uniqueness score for 5-HT_2C protein for each model

5-HT2C	Pocket2Mol	PocketFlow	Lingo3DMol	DiffSBDD	PMDM
Total ligands	510	500	192	499	500
Validity2D	1.00	1.00	1.00	0.97	1.00
Uniqueness	1.00	0.96	0.95	1.00	1.00
Time (min)	7.08	19.61	184.14	35.50	192.35

Table 5.

Total ligands generated, Validity2D, and Uniqueness score for DRD2 protein for each model

DRD2	Pocket2Mol	PocketFlow	Lingo3DMol	DiffSBDD	PMDM
Total ligands	500	500	145	500	500
Validity2D	1.00	1.00	1.00	0.98	1.00
Uniqueness	1.0	0.96	0.83	1.00	1.00
Time (min)	5.12	20.35	186.19	29.20	297.33

Across both GPCR targets, 5-HT_2C and DRD2, most models exhibited similar distributions of molecular weight (MW), log P, synthetic accessibility (SA Score) and QED between 5-HT_2C and DRD2 (Fig. 2). Pocket2Mol was a notable exception, showing higher mean QED values for DRD2 compared to 5-HT_2C. PMDM and Chemistry42 generated ligands with the narrowest MW distributions, centred around 400 Da. Meanwhile, other models produced ligands spanning a broader MW distribution ranging from approximately 100 to 700 Da (Fig. 2A), suggesting more consistent physicochemical profiles for PMDM and Chemistry42. The log P distributions revealed substantial differences among models. While many generated ligands fell below the commonly accepted threshold of log P < 5, PocketFlow generated a large fraction of hydrophobic ligands, whereas Lingo3DMol favoured highly hydrophilic compounds (Fig. 2B). These trends deviate from the physiochemical characteristics of known GPCR ligands such as ritanserin and haloperidol.

Fig. 2.

Each model has varying distribution of physicochemical properties. Violin and box plot shows the physicochemical properties distribution of A molecular weight (MW), B log P, C SA Score, and D QED for generated ligands targeting GPCRs

Comparison of MW and SA Score distribution showed that smaller ligands generated by Pocket2Mol and PocketFlow tend to have lower SA scores (Fig. 2C). In contrast, smaller ligands generated by diffusion-based models (Fig. 2A) exhibited higher SA Scores than Lingo3DMol (Fig. 2C), suggesting that diffusion-based models generate structurally more complex scaffolds even at lower molecular weights. Overall, all models covered a broad QED range from 0.1 to 1.0, with PMDM and Chemistry42 exhibited better mean QED values and narrower distribution than the rest of the models (Fig. 2D). Taken together, Chemistry42 achieved the most balanced performance across physicochemical properties, combining relatively high QED with moderate SA Score.

Docking and pose stability for GPCR ligands

Docking analysis revealed that generated conformations from all models generally did not align well with poses obtained from docking programs. Redocking using Glide SP and Vina resulted in RMSD exceeding above 2 Å for a large fraction of ligands, including those generated by commercial Chemistry42 software (Fig. 3A). Glide SP “in-place” docking failed for many compounds except Chemistry42, yielding positive Coulumb-van der Waals energies (Supplementary Fig. 8). These observations were consistent with Validity3D results, which showed poor geometric quality for most models except Chemistry42, unless MMFF94 forcefield minimisation was applied (Supplementary Fig. 9).

Fig. 3.

Few generated ligands mimic 5-HT_2C and DRD2 agonists. A The violin and box plot of redocked glide docking score, redocked vina docking score, glide redocked RMSD, and vina redocked RMSD. B, C Docking pose of best ligand of each model against 5-HT_2C (B) and DRD2 (C) with reference to ritanserin and haloperidol, respectively. Purple dashed line indicates polar interaction or salt bridge

Following redocking, only a small number of ligands achieved docking scores better than the reference GPCR ligands ritanserin and haloperidol using Glide SP and Vina (Fig. 3). While Vina identified several Pocket2Mol generated ligands with favourable docking scores against 5-HT_2C (Fig. 3A), these compounds contained highly unusual polycyclic fused ring systems and were rejected by Glide SP docking (Supplementary Fig. 10). Vina also predicted multiple ligands outperforming haloperidol for DRD2 protein across most models except Lingo3DMol (Fig. 3A).

Despite these limitations, all models were able to generate ligands that recapitulated key pharmacophoric interactions of both ritanserin and haloperidol with 5-HT_2C (Fig. 3B) and DRD2 (Fig. 3C), respectively. Selected ligands contained a protonated nitrogen forming an ionic interaction with the conserved aspartate residue and a hydrophobic moiety engaging W324 in 5-HT_2C (Fig. 3B) and W100 in DRD2 (Fig. 3C). Several models also reproduced fluorophenyl-like moieties (Fig. 3B) observed in reference ligands, suggesting partial learning of pocket-specific interaction patterns. However, the proportion of ligands that satisfy these interaction criteria in combination with acceptable drug-like properties was very low. For Pocket2Mol, the best ligands identified for both GPCRs were fragments (Fig. 3B,C), despite the model generating many compounds with MW over 500 Da (Fig. 2A). Even Chemistry42 achieved a virtual hit rate of only ~ 3% (Table 6), and approximately half of these hits exhibited Glide redocking RMSD values greater than 3 Å (Fig. 3A). These results indicate that majority of the generated GPCR ligands are unfavourable in terms of structure, conformations, or physicochemical properties (Table 7).

Table 6.

Total number of virtual hits and properties of the chosen best ligand for each model against 5-HT_2C

Model name	Total virtual hits	Chosen best ligand
		ID	QED	log P	Phase Score	Glide/ kcal mol−1	RMSD / Å	Vina / kcal mol−1
Pocket2Mol	0	num500_389	0.573	1.394	–0.531	–6.168	3.750	–5.382
PocketFlow	1	num500_450	0.730	4.226	–0.114	–8.072	6.713	–9.325
Lingo3DMol	0	num500_8	0.793	1.279	0.561	–7.746	1.069	–7.407
DiffSBDD	15	num500_49	0.565	4.999	1.297	–8.989	1.852	–10.746
PMDM	3	num500_263	0.667	1.962	0.217	–8.033	1.621	–12.351
Chemistry42	16	num500_72	0.661	3.518	0.608	–8.835	1.277	–10.025

RMSD corresponds to the redocking RMSD between generated conformation and redocking Glide SP. Virtual hits are defined by ligands with QED > 0.5 and Glide docking of –8 kcal mol⁻¹ with nonzero phase score

Table 7.

Total number of virtual hits and properties of the chosen best ligand for each model against DRD2

Model Name	Total virtual hits	Chosen best ligand
		ID	QED	log P	Phase Score	Glide/ kcal mol−1	RMSD / Å	Vina / kcal mol−1
Pocket2Mol	0	num500_109	0.817	–0.062	–0.870	–7.045	7.025	–8.022
PocketFlow	2	num500_198	0.512	7.537	0.185	–8.868	2.330	–11.218
Lingo3DMol	1	num500_75	0.580	0.980	0.318	–8.451	4.710	–9.695
DiffSBDD	17	num500_149	0.820	3.970	0.939	–8.201	1.697	–8.709
PMDM	15	num500_285	0.740	5.381	0.405	–9.372	2.746	–10.817
Chemistry42	9	num500_465	0.683	3.077	0.367	–8.481	0.875	–9.698

RMSD corresponds to the redocking RMSD between generated conformation and redocking Glide SP. Virtual hits are defined by ligands with QED > 0.5 and Glide docking of –8 kcal mol⁻¹ with nonzero phase score

Generative model performance on kinase targets

The findings for kinase targets largely mirrored trends observed for GPCRs with respect to model performance (Tables 8 and 9) and physicochemical properties (Fig. 4). A notable distinction was that kinases-targeted ligands were generally more polar than GPCR-targeted ligands (Fig. 4B), indicating that models generated ligands with similar overall properties despite the differences in the protein family or protein class.

Table 8.

Total ligands generated, Validity2D, and Uniqueness score for CDK2 protein for each model

CDK2	Pocket2Mol	PocketFlow	Lingo3DMol	DiffSBDD	PMDM
Total ligands	517	500	330	500	500
Validity2D	1.0	1.0	1.0	0.97	1.0
Uniqueness	1.0	0.96	0.98	1.0	1.0
Time (s)	5.38	18.90	188.76	22.97	122.43

Table 9.

Total ligands generated, Validity2D, and Uniqueness score for GSK3β protein for each model

GSK3β	Pocket2Mol	PocketFlow	Lingo3DMol	DiffSBDD	PMDM
Total ligands	500	500	480	500	500
Validity2D	1.0	1.0	1.0	0.94	1.0
Uniqueness	1.0	0.92	0.98	1.0	1.0
Time (s)	4.29	20.91	270.72	25.76	128.17

Fig. 4.

Kinase tested has consistent physicochemical properties with CDK2 and GSK3β. Violin and box plot consisting of A molecular weight, B log P, C SA Score, and D QED for kinase generated ligands

Generated ligands showed lower redocking RMSD values targeting kinases relative to GPCRs, despite the larger binding pockets of kinases. RMSD distributions were narrower (within 15 Å) compared to GPCRs (20 Å) (Fig. 5A), suggesting that trained models are better suitable for kinase targets than GPCRs. This observation correlates with the higher representation of unique kinase structures available in the CrossDocked2020 dataset compared to GPCRs (Supplementary Fig. 11). Furthermore, a subset of generated ligands was found to dock outside the CDK2 ATP-binding pocket (Supplementary Fig. 12).

Fig. 5.

Few generated ligands mimic CDK2 and GSK3β inhibitors. A The violin and box plot of redocked glide docking score, redocked vina docking score, glide redocked RMSD, and vina redocked RMSD. B, C Docking pose of best ligand of each model against CDK2 (B) and GSK3β (C) with reference to molecule 13 and molecule 18, respectively. Purple dashed line indicates polar interaction or salt bridge

For CDK2, the best-generated molecules reproduced hinge-binding interactions via amino pyrazole or related moieties similar to reference CDK2 inhibitor ‘molecule 13’ (Fig. 5B). Pocket2Mol generated a five-fused-ring moiety containing an amino pyrazole, whereas Lingo3DMol and Chemistry42 produced amino pyrazolopiperidine scaffolds for the hinge region (Fig. 5B). DiffSBDD lacked another nitrogen atom in its 3-imidazolin-5-one moiety, preventing hydrogen bonding to the L83 backbone carbonyl.

For GSK3β, both Glide SP and Vina predicted multiple generated ligands with docking scores more favourable than those of the reference ligand in silico (Fig. 5A). Both ‘molecule 18’ and selected best molecules consistently formed two hinge hydrogen bonds (Fig. 5C). While PocketFlow uniquely employed a pyridine moiety to interact with K85, PMDM and Pocket2Mol used carbonyl groups as hydrogen bond acceptors to interact with K85 (Fig. 5C). Similar to CDK2, the chosen generated ligand has a moiety containing N–C–N group to bind to the hinge region: PocketFlow has a 7-azaindole moiety, Chemistry42 has a 7-amino-1H-indazole moiety, and DiffSBDD has a benzimidazolinone moiety. Interestingly, DiffSBDD generated a benzimidazolinone moiety resembling a cyclic amide of staurosporine (Supplementary Fig. 13). Unlike GPCRs, all selected kinase ligands satisfied the virtual hit criteria (Tables 10 and 11).

Table 10.

Total number of virtual hits and properties of the chosen best ligand for each model against CDK2

Model name	Total virtual hits	Chosen best ligand
		ID	QED	log P	Phase Score	Glide/ kcal mol−1	RMSD / Å	Vina / kcal mol−1
Pocket2Mol	28	num500_192	0.700	1.039	0.514	–9.295	2.688	–10.002
PocketFlow	1	num500_166	0.749	4.823	0.502	–9.264	4.712	–10.100
Lingo3DMol	21	num500_150	0.649	4.588	0.667	–9.025	1.766	–10.223
DiffSBDD	20	num500_137	0.779	0.649	0.729	–8.932	1.886	–8.093
PMDM	9	num500_21	0.535	2.744	0.543	–8.443	3.946	–9.328
Chemistry42	24	num500_148	0.731	3.480	0.874	–10.004	0.512	–10.575

RMSD corresponds to the redocking RMSD between generated conformation and redocking Glide SP. Virtual hits are defined by ligands with QED > 0.5 and Glide docking of –8 kcal mol⁻¹ with nonzero phase score

Table 11.

Total number of virtual hits and properties of the chosen best ligand for each model against GSK3β

Model name	Total virtual hits	Chosen best ligand
		ID	QED	log P	Phase Score	Glide/ kcal mol−1	RMSD / Å	Vina / kcal mol−1
Pocket2Mol	35	num500_150	0.839	1.990	0.657	–8.764	4.966	–9.605
PocketFlow	8	num500_322	0.602	4.363	0.532	–8.499	5.158	–8.433
Lingo3DMol	2	num500_171	0.658	–0.011	–0.213	–8.292	7.603	–8.099
DiffSBDD	12	num500_42	0.704	1.615	0.608	–10.112	1.457	–9.826
PMDM	19	num500_184	0.737	4.643	0.439	–9.471	3.971	–10.398
Chemistry42	7	num500_331	0.668	5.239	0.571	–8.084	2.502	–10.638

RMSD corresponds to the redocking RMSD between generated conformation and redocking Glide SP. Virtual hits are defined by ligands with QED > 0.5 and Glide docking of –8 kcal mol⁻¹ with nonzero phase score

Generative model performance on hydrolase targets

Finally, we evaluated model performance on two hydrolases: SARS-CoV-2 3CLpro and NA. PMDM generated only 212 ligands in 2 h due to a bond order overflow error (Table 12), and Pocket2Mol also failed to generate 500 ligands for NA, even with relaxed parameters (Table 13).

Table 12.

Total ligands generated, Validity2D, and Uniqueness score for 3CLpro protein for each model

3CLpro	Pocket2Mol	PocketFlow	Lingo3DMol	DiffSBDD	PMDM
Total ligands	513	500	426	500	212
Validity2D	1.0	1.0	1.0	0.99	1.0
Uniqueness	1.0	0.96	0.59	1.0	1.0
Time (s)	4.56	17.95	180.15	22.82	124.67

Table 13.

Total ligands generated, Validity2D, and Uniqueness score for NA protein for each model

NA	Pocket2Mol	PocketFlow	Lingo3DMol	DiffSBDD	PMDM
Total ligands	–	500	100	500	500
Validity2D	–	1.0	1.0	0.98	1.0
Uniqueness	–	0.96	0.45	1.0	1.0
Time (s)	–	25.23	201.24	19.78	205.21

The physicochemical property distributions for 3CLpro resembled those of previous protein targets. Unlike 3CLpro, generated ligands targeting NA have a lower mean MW (Fig. 6A), higher mean log P (Fig. 6B), and poorer mean QED values across models, except Chemistry42 (Fig. 6D). PMDM generated ligands targeting NA also showed lower SA scores than those targeting 3CLpro (Fig. 6C). This indicates that NA presents a particularly challenging target to generate potential virtual hits for these models.

Fig. 6.

Generated ligands targeting NA have different physicochemical properties profile compared to other proteins. Violin and box plot consisting of A molecular weight, B log P, C SA Score, and D QED for hydrolase generated ligands. The horizontal lines in between proteins for each model shows the mean of the respective value

Although the reference ligand GC376 is a covalent inhibitor, we prompted models to generate non-covalent ligands and evaluated them on non-covalent docking against 3CLpro. As expected, redocking GC376 yielded RMSD values exceeding 2 Å (Fig. 7A). Although both Glide SP and Vina redocking suggested ligands with docking scores better than the reference (Fig. 7A), very few ligands passed the virtual hit filters for best molecules for 3CLpro (Table 13). Selected ligands formed fewer hydrogen bonds than GC376, except for ligand generated by Lingo3DMol (Fig. 7B), and frequently interacted with F140 residue rather than the canonical E166 and H163 residues (Fig. 7B).

Fig. 7.

Very few generated ligands mimic 3CLpro and NA inhibitor. A The violin and box plot of redocked glide docking score, redocked vina docking score, glide redocked RMSD, and vina redocked RMSD. B, C Docking pose of best ligand of each model against 3CLpro (B) and NA (C) with reference to GC376 and oseltamivir, respectively. Purple dashed line indicates polar interaction or salt bridge

In contrast, all models demonstrated partial understanding of NA binding interactions. Autoregression models and DiffSBDD reproduced the carboxylate interactions of oseltamivir with three arginine residues (Fig. 7C), while other models engaged additional residues such as E119 and R152. However, several models failed to reproduce oseltamivir’s pentyl tail interaction with E276 in the H274Y mutant (Fig. 7C). Owing to generally poor docking scores, only a small number of NA ligands satisfied the cutoff (Tables 14, 15).

Table 14.

Total number of virtual hits and properties of the chosen best ligand for each model against 3CLpro

Model name	Total virtual hits	Chosen best ligand
		ID	QED	log P	Phase score	Glide/ kcal mol−1	RMSD / Å	Vina / kcal mol−1
Pocket2Mol	0	num500_213	0.589	3.569	N/A	–7.726	7.941	–6.194
PocketFlow	0	num500_262	0.791	3.902	0.188	–7.277	5.228	–7.976
Lingo3DMol	1	num500_274	0.507	2.187	0.188	–8.548	5.391	–6.633
DiffSBDD	0	num500_386	0.504	2.101	0.440	–7.063	1.482	–6.017
PMDM	0	num500_18	0.684	3.668	0.133	–6.465	6.405	–7.140
Chemistry42	0	num500_285	0.686	4.149	0.464	–7.597	5.935	–7.200

RMSD corresponds to the redocking RMSD between generated conformation and redocking Glide SP. Virtual hits are defined by ligands with QED > 0.5 and Glide docking of –8 kcal mol⁻¹ with nonzero phase score

Table 15.

Total number of virtual hits and properties of the chosen best ligand for each model against NA

Model name	Total virtual hits	Chosen best ligand
		ID	QED	log P	Phase score	Glide/ kcal mol−1	RMSD / Å	Vina / kcal mol−1
Pocket2Mol	0	num100_97	0.460	0.502	0.606	–8.514	2.000	–5.738
PocketFlow	0	num500_132	0.622	2.421	0.549	–4.789	5.590	–6.325
Lingo3DMol	1	num500_75	0.517	–2.483	0.437	–9.227	2.133	–7.034
DiffSBDD	1	num500_313	0.519	0.024	0.848	–8.289	1.344	–7.370
PMDM	0	num500_449	0.525	–0.819	0.669	–4.921	5.219	–8.552
Chemistry42	0	num500_188	0.769	1.068	0.753	–5.687	1.064	–8.008

RMSD correspond to the redocking RMSD between generated conformation and redocking Glide SP. Virtual hits are defined by ligands with QED > 0.5 and Glide docking of –8 kcal mol⁻¹ with nonzero phase score

Discussion

Generative models can generalise chemical structure and valency

This study evaluated whether representative structure-based generative models could design effective virtual hit candidates across six clinically relevant protein targets. Across all models, we observed a strong ability in understanding the ligand valency, reflected by minimal parsing errors and the capacity to generate up to 500 unique ligands. This outcome is expected given the size and diversity of training datasets used by both autoregressive and diffusion-based models, as well as the widespread use of transformer-based architectures capable of encoding atom types, bond types and local chemical environments. However, correct valency alone is insufficient for productive hit discovery. While most models can generate chemically valid molecules at scale, downstream analysis revealed substantial limitations in scaffold quality, conformational realism, and binding plausibility, underscoring the gap between chemical validity and medicinal relevance.

HamDiv and unique generic BM scaffold metrics differentiate models with diverse and non-diverse ligands

Our benchmark study demonstrates that HamDiv and the number of unique Generic BM scaffolds provide clearer differentiation between autoregressive and diffusion-based models. These two metrics addressed the issue that adding few substituents to ligand scaffold, akin to how autoregressive model works, increases ligand diversity significantly [30]. Hence, it is expected that autoregressive model would perform the least than other models. Diffusion-based models, particularly DiffSBDD, generated ligands with higher HamDiv scores throughtheir ability to randomise the atom counts for each ligand during generation [10]. However, when faced with low atom counts, DiffSBDD may generate aliphatic ligands which can contribute to a small number of generic BM scaffolds. In contrast, PMDM produced large number of unique generic BM scaffolds despite lower HamDiv scores. This is largely attributable to its fixed heavy atom count of 29 [2], which biases the model toward compact, highly fused polycyclic structures that readily generate distinct scaffolds. PocketFlow occasionally outperformed diffusion models on specific targets, suggesting that its flow-based architecture may better preserve scaffold diversity observed in the training dataset. Chemistry42, which integrates multiple generative models, unsurprisingly yielded the most heterogeneous ligand set.

As both quantity and quality (i.e., diversity) influence HamDiv score, benchmarking the number of generated ligands could measure the capability of each model. Existing methods use Average Tanimoto ECFP Distance and other similar variations to measure diversity [2, 6, 10]. Importantly, HamDiv MCES and HamDiv ECFP both outperformed average pairwise Tanimoto distance in distinguishing meaningful diversity. While HamDiv MCES offers high structural fidelity, it is computationally expensive. On the other hand, HamDiv ECFP calculation is rapid but compromised with their accuracy. Together these metrics better capture diversity relevant to medicinal chemistry decision-making. Interestingly, we observed that diversity metrics remained relatively consistent across different targets at a given sample size. This suggest that current generative models may be more strongly influenced by training data distributions and model architecture than by subtle differences in pocket geometry. In addition, the lack of explicit modeling of protein flexibility or induced fit may further limit the ability of these models to adapt diversity to target-specific structural constraints. This observation highlights a potential limitation of current structure-based generative approaches in capturing fine-grained differences between binding environments.

Generative model tends to generate unrealistic fused ring system

A consistent failure mode across both autoregressive and diffusion-based models was the generation of chemically implausible fused ring systems, in some cases containing six to eight fused rings. Pocket2Mol is particularly prone to this behaviour, producing ligands that were either too fragmented or excessively complex. This can be attributed to the increase in iteration and decrease in beam size intended to generate more diverse ligands, forcing Pocket2Mol to fit larger molecules in small protein pocket. This observation helps reconcile conflicting reports in the literature regarding Pocket2Mol performance [4, 6]. Diffusion models also exhibited similar tendencies, with PMDM generating more fused rings than DiffSBDD due to its fixed atom count constraint. This constraint forced the ligands to have a highly compact structure with multiple fused ring system to fit into small protein pocket. Additionally, because bond inference is performed through Openbabel, the final ligand structure may not reflect the intended atomic geometry. These design choices likely contribute to the prevalence of unrealistic polycyclic scaffolds.

PocketFlow appears less affected by this issue, likely due to its incorporation of chemical domain knowledge within the generation process. Lingo3DMol, which constructs molecules fragment-by-fragment rather than atom-by-atom, also avoids extreme ring fusion, suggesting that fragment-level priors may be beneficial. While fragment diffusion remains challenging, future diffusion models could mitigate this issue by incorporating chemically informed bond representations, as explored in approaches such as DiffGui [17], or introducing loss functions that penalize excessive ring fusion.

Generated ligand conformation rarely match docking poses

Across all targets and models, generated ligand conformation showed poor agreement with redocked poses, even after force-field minimization. Wide RMSD distribution before and after minimization indicate that the generated conformations are generally inconsistent. Notably, Chemistry42 also showed similar RMSD distribution despite scoring the highest Validity3D score of above 0.5. This suggest that conformational validity metrics alone may overestimate docking pose quality, particularly if internal minimisation is applied during generation [20].

However, the above finding does not align with Lingo3DMol study. The discrepancy between our findings and the reported performance of Lingo3DMol with an RMSD of 0.9 may stem from methodological differences. We only used the best Glide SP docking pose as a reference, whereas Feng et al. used multiple docking conformations [6]. Nevertheless, this highlights the importance of reporting both redocking RMSD and docking scores for transparent benchmarking. Among the other models, only Lingo3DMol reported such metrics explicitly [6].

The frequent failure of Glide SP “in-place” docking prior to forcefield minimisation further suggests that many generated conformations initially clash with the protein pocket, leading to unfavourable Coulumb-van der Waals energy. While minimisation can resolve these clashes, the underlying issue remains, as current generative models do not explicitly learn force-field-compatible binding conformations. Whether integrating physics-aware objectives or hydrogen bond specific constraints can improve pose realism remains an open question, as previously noted by Fan et al. [1].

Virtual hit rate remains low across all models

A key finding of this study is the consistently low hit rates across all models when multiple drug-like criteria are applied simultaneously. While individual metrics such as docking score or QED may appear favourable in isolation, only a small fraction of generated ligands satisfy pharmacophore matching, docking and drug-likeness criteria concurrently. Pocket2Mol achieved comparatively higher virtual hit rates for kinases. However, these hits have highly similar scaffold, limiting their practical utility. In hindsight, integrating our diverse metrics, such as the number of unique Generic BM scaffolds into the virtual hit rate would provide a more robust assessment of model performance. DiffSBDD, PMDM, and Chemistry42 performed modestly overall, but virtual hit rates across all targets remained below 5%. These values are consistent with previously reported results, including the 0.2% hit rate reported for PocketFlow [19] and the sparse hits generated by PMDM against 3CLpro [2]. These findings emphasise that reporting a single “best molecule” is insufficient to characterize model performance. Instead, virtual hit rate provides a more representative measure of a model’s ability to generate actionable candidates.

Generative models design kinase ligands more effectively than GPCRs

Across the evaluated targets, generative models consistently produced higher virtual hit rates and lower redocking RMSD values for kinases than for GPCRs. These findings can be attributed to several factors. First, kinase structures are substantially overrepresented in the CrossDocked2020 dataset, reflecting the availability of high-quality kinase crystal structures. This abundance likely facilitates model generalisation of conserved ATP-binding site features, particularly hinge region interactions, which are shared across many kinases. This effect is exemplified by DiffSBDD, where the selected ligand contains a moiety resembling the canonical kinase inhibitor staurosporine. Second, redocking RMSD is highly sensitive to ligand orientation. For GPCRs, which typically possess narrow and elongated binding pockets, ligands adopting a 180° flipped orientation relative to the docked pose could incur large RMSD penalties despite maintaining similar interaction patterns [41, 42]. In contrast, kinase ligands with high RMSD values are more frequently observed to dock outside the ATP-binding pocket. Together, these factors suggest that both dataset bias and pocket geometry contribute to the overall apparent performance of generative models on kinase targets relative to GPCRs.

SARS-CoV-2 3CLpro remains a challenging target

Among all targets evaluated, SARS-CoV-2 3CLpro proved particularly difficult. Except for Lingo3DMol, none of the models generated ligands that satisfied the virtual hit criteria. Several factors likely contribute to this outcome. First, the reference ligand GC376 is a covalent molecule, whereas models were evaluated under non-covalent docking conditions. Structural differences between covalent and non-covalent inhibitor bound conformations may hinder pharmacophore transfer, particularly the inability to redock WU-04 with the correct conformation based on the crystal structure [40] (Supplementary Fig. 14).

Second, 3CLpro itself was absent from the CrossDocked2020 training set [18]. However, analogues of the SARS-CoV-2 3CLpro were included, such as the SARS-CoV 3CLpro, SARS-CoV PLpro, and the MERS-CoV 3CLpro [18]. Despite the presence of these related viral proteases in the training set, these homologues share structural, subtle differences in pocket architecture and interaction patterns may prevent effective generalization. This raises the possibility that current models struggle to generalise across viral protease families. Finally, 3CLpro active site comprises multiple subsites (S1, S2, S4, and S1’) [43] with diverse polar interactions, making it challenging for generative models to simultaneously satisfy all binding requirements. The high RMSD values observed for the selected ligands support this interpretation (Table 14). Furthermore, many reported 3CLpro inhibitors rely on covalent warheads and peptidomimetic scaffolds to engage the catalytic cysteine and multiple subsites [44, 45] features that are not explicitly captured in non-covalent generative modeling frameworks.

Best generated ligands for neuraminidase (NA) remain fragment-like

Oseltamivir binds within a relatively a small pocket of NA. However, structural inspection reveals an adjacent cavity that could potentially be accessed by larger ligands if interface water molecules were displaced. We therefore evaluated whether generative models could produce ligands that extend beyond the canonical Oseltamivir binding pocket while retaining key oseltamivir pharmacophores. Although many generated ligands exhibited higher MW supported the possibility of dual pocket occupancy, these compounds were excluded from virtual hits as they failed to satisfy the pharmacophore criteria. One possible explanation is that the adjacent pocket lies beyond the defined protein–ligand distance cutoff used during pocket extraction, resulting in incomplete representation of this region during ligand generation. Consequently, residues critical for stabilizing dual-pocket binding may not have been available to guide generation. This effect may also explain why Lingo3DMol predominantly generated lower MW ligands localized within the oseltamivir pocket, despite its architecture favouring to identify polar anchor residues. In addition, a large portion of generated ligands failed to meet the QED or docking score cutoffs, particularly among compounds larger than fragment size. Because the virtual hit criteria jointly required acceptable drug-likeness and good pharmacophore matching, ligands that satisfied these filters were biased towards smaller, fragment-like molecules.

Limitations and future work

This study has several limitations. First, statistical comparisons were not performed due to the limited number of protein targets. Expanding the benchmark to include additional targets would enable formal statistical analyses, such as two-way ANOVA. Second, most of the protein targets used were present in the training data instead of unseen data, potentially inflating apparent performance. Notably, models performed poorly even on these familiar targets, suggesting fundamental limitation in protein–ligand generalization. Recent work by Liu et al. evaluated pocket-based generative models across a larger and diverse set of protein structures [46]. While such large-scale benchmarks are valuable for identifying general performance trends, our study focuses on a detailed, multi-dimensional evaluation incorporating diversity, pharmacophore matching, and pose reproducibility. Direct comparison of model rankings across different benchmarking frameworks would be an interesting direction for future work, particularly to assess the consistency of conclusions across evaluation criteria and datasets. Future studies should therefore evaluate generative models on recently resolved protein–ligand complexes that are absent in the training datasets, ideally from underrepresented protein classes, to better assess generalization performance.

In addition, this study does not directly compare structure-based generative models with virtual screening approaches over ultra-large, synthetically accessible libraries such as EnamineREAL or Freedom Space. While generative models have the potential to explore broader chemical space beyond predefine libraries, synthetic tractability remains a key challenge. In contrast, synthesis-on-demand libraries offer high feasibility and well-established acquisition pipelines. A systematic comparison between these approaches would provide valuable insights into their relative strengths and practical applicability and represents an important direction for future benchmarking studies.

Finally, we generated up to 500 ligands per target per model, which may not fully capture the tail of the underlying metric distributions. This may partially explain discrepancies between our findings and those reported in original model publications, which often sample substantially larger number of molecules. For wet-lab validation, while some studies generate over 100,000 ligands prior to hit selection [2, 10], such scales are often impractical for routine benchmarking. Future works could explore how virtual hit rate scale with sample size and whether early enrichment is predictive of downstream success.

Conclusion

Taken together our results indicate that current structure-based generative models excel at producing chemically valid molecules but remain limited in their ability to generate conformationally realistic, drug-like ligands that reliably engage protein targets. By introducing ScrambleBench and emphasizing diversity-aware metrics, pose realism, and virtual hit rate, this study provides a more stringent and medicinal chemistry relevant framework for evaluating generative models. We anticipate that these findings will inform the development of next-generation generative models that better integrate reward function for drug-like properties, physical constraints, and target-specific context. We also encourage future benchmark studies to include diversity within models shown such as unique BM scaffold or HamDiv metric and defining their own virtual hit rate. We believe by incorporating these improvements, generative models have huge potential to complement structure-based drug design and to explore unknown chemical space to design drugs against therapeutic targets.

Author contributions

Veincent Yap, Padmanabhan Anbazhagan and Weijun Xu conceived computational modeling experiment and wrote the manuscript. Veincent Yap prepared all the figures with guidance from Pan Xu. Pan Xu, Frankie Y. Mak, Klement Foo, and Congbao Kang reviewed and edited the manuscript. All authors reviewed the manuscript.

Funding

This work is partially supported by Singapore PREPARE grant (CS1-2025-016).

Data availability

All original data is available on the provided Zenodo public repository with the link https://zenodo.org/records/18503149 and Github https://github.com/Feriolet/ScrambleBench.

Declarations

Competing interests

Author Xu Weijun is a member of the Editorial Board for Journal of Cheminformatics, and was not involved in the journal’s review of, or decisions related to, this manuscript.