Efficient and valid large molecule generation via self-supervised generative models

Abstract

The realm of molecular design, particularly for large molecules, presents unique challenges and opportunities in drug discovery and materials science. Large molecule design is inherently more complex and less explored compared to designing small molecules, adding significant difficulty in generative modeling. We aim to establish strong baselines for better scalability, efficiency, and generative performance in this domain. We evaluate the scalability and performance of generative AI models, initially effective for small molecule design, in generating large molecules for potential drugs in gene-based therapies, immunotherapies, hormonal regulators, and targeted cancer therapies. Our findings indicate that computational strategies and model architectures designed for small molecules may not readily extend to large molecular structures. To address these limitations, we explore masked language modeling strategies alongside advanced tokenization methods, including Atom-Pair Encoding (APE), to enhance generative AI models. We probe how incorporating such strategies, particularly the APE tokenization method that explicitly captures structural and chemical characteristics, can significantly improve design capabilities for complex molecular structures. Overall, our results demonstrate both the potential and challenges of deep generative modeling for large molecules and how the proposed enhancements may bridge the gap in generating large molecules when novel discovery is the ultimate goal.

Introduction

The design and synthesis of molecules, especially large molecules, is a foundational challenge within molecular science with significant implications across various fields such as drug discovery. The vast array of potential molecular configurations, estimated to be in the order of ≈10^{60 1}, presents a formidable challenge, making it nearly impossible for human experts to exhaustively enumerate and experimentally validate each potential molecule in traditional laboratory settings. This underscores the critical need for innovative computational approaches that can extend beyond the limitations of human labor capacity and conventional methods.

In the face of this challenge, the recent surge in artificial intelligence and machine learning (AI/ML) advancements, especially in generative AI methods, has marked the dawn of a new epoch in generative molecular design (GMD). These AI/ML models, with their remarkable ability to learn and predict, have propelled the field forward, enabling the creation of candidate molecules with efficiency and effectiveness previously unattainable. However, most existing literature, including studies like refs. ^2,3, and ref. ⁴, primarily focuses on small molecules, often not addressing the unique challenges associated with the generation of large molecules. This suggests a gap in research that our current study aims to address. By testing the scalability of models that have demonstrated success in small molecule datasets^2–7, we aim to bridge this gap and extend their applicability to the more challenging task of large molecule design.

The generation of large molecules is inherently complex due to the nuanced nature of their stability and the intricate interplay of their structural components. For instance, large molecules often exhibit a higher degree of conformational flexibility and may possess reactive functional groups that can complicate synthesis and stability. These characteristics present not just a computational modeling challenge, but also a fundamental chemical synthesis puzzle that must be addressed to enable the practical realization of large molecule designs.

Drawing from the pivotal insights from the GMD literature⁸, we recognize the crucial role of choosing the appropriate input data representations when training these generative AI models—be it 1D strings, 2D graphs, or 3D geometries—each with its unique benefits and computational implications. Our exploration is centered on the 1D string representations of molecules, employing the widely adopted SMILES (Simplified Molecular Input Line Entry System)⁹ and SELFIES (Self-referencing Embedded Strings)¹⁰ formats. Despite existing recurrent neural network (RNN) or transformer-based models, including large language models (LLMs) trained on extensive molecule corpora, being well-suited for such 1D string representations, we recognized early on that the direct application of these models to large molecules faces significant challenges. This is notably due to the more complex and varied structures that large molecules present, which are often more naturally represented in 2D and 3D forms. Graph-based models, which use these forms, suffer from a quadratic dependence on time complexity and memory requirement with respect to the feature dimension and mini-batch size⁷. To date, there are no reported graph-based models trained on molecules containing more than 100 atoms, highlighting a critical gap in the field for large molecule generation.

In response to these challenges, we develop and evaluate generative models specifically for large molecule design by optimizing them with corresponding 1D input string representations. Our methodological innovations include: (1) implementing self-supervised learning with molecular generative models to enable masked sequence prediction for better learning latent structural patterns in large molecules; (2) introducing Atom-Pair Encoding (APE)¹¹, a novel tokenization method, into molecular generative models for improving scalability, efficiency, and validity of large molecule generation; (3) integrating different string representations and evaluating their corresponding generative performances, focusing on large molecules; (4) strategic modifications of generative models to dynamically manage memory by deleting and overwriting unused model parameters during training, further enhancing computational efficiency and model scalability. By subjecting these developments to rigorous testing against established evaluation metrics provided by the MOSES¹² platform, we aim to not only assess their ability to generate valid, unique, and structurally diverse large molecules but also to pave the way for substantial advancements in the field of molecular generative modeling. Our study seeks to contribute to generative AI in drug discovery, specifically by addressing the unmet need for effective computational strategies tailored to the generation of large, complex molecular structures.

The following sections will detail the specific methodologies employed, the datasets used, and the results obtained from this comparative study, ultimately contributing to the broader understanding of the capabilities and limitations of generative AI in the realm of large molecule design.

Results

We conduct performance evaluation experiments on large molecule design focusing on the described string-based generative models, evaluated comprehensively using an array of metrics that gauge the validity, uniqueness, novelty, and structural integrity of the generated large molecules.

Datasets

We have collected two datasets from public databases for our performance evaluation tasks:

ChEMBL-Large

This includes a subset of the ChEMBL database, where we have collected the molecules with an average molecular weight of 1799 Da. ChEMBL¹³ is renowned for its collection of bioactive molecules with drug-like properties. This dataset is pivotal in training our models to handle large molecules and to evaluate the capabilities of large molecule design.

MOSES

Derived from the ZINC database¹⁴, MOSES¹² is a benchmarking platform for molecular generation tasks. This dataset comprises molecules with an average molecular weight of 307 Da, representing a database of three-dimensional chemical structures.

To evaluate the generalizability of generative molecular design methods, we have further combined these two datasets into the following mixed dataset:

ChEMBL-Large + MOSES

This mixed dataset combines the complexities of both ChEMBL-Large and MOSES, aiming to capture a broad spectrum of molecular sizes and complexities. It allows us to train our models on a diverse range of molecular structures, from small to large molecules.

Evaluation metrics

The evaluation of generative models in this research is performed using the pipeline in the MOSES¹² platform, which leverages a suite of metrics to assess various aspects of the generated molecules. These metrics are calculated using the RDKit library, which provides chemoinformatics and machine learning tools for molecular data. We evaluate generated molecules using the following metrics:

Fraction of valid and unique molecules

This set includes the validity metric (Valid) quantifies the percentage of chemically valid molecules among the generated SMILES strings. Validity is determined by RDKit to parse the strings and verify their structural integrity. Besides, the uniqueness (Unique@k) measures the diversity of generated molecules, ensuring that the generative model produces a variety of chemical structures, not just variations of a few templates.

Novelty

This metric (Novelty) captures the ability of the model to generate molecules that are not present in the training set, indicative of the model’s creativity and its potential to contribute novel compounds to the chemical space.

Filters

The Filters metric (Filters) provides the proportion of generated molecules that pass the same filters applied during the construction of the training dataset, including medicinal chemistry filters (MCF) and PAINS filters, which screen out molecules likely to interfere with biological assays.

Internal diversity

This IntDivp metric evaluates the chemical diversity within the generated set of molecules. A higher value of internal diversity suggests a broad coverage of the chemical space, reducing the likelihood of mode collapse where the model would only generate a narrow range of molecules. IntDivp2 is similar to IntDivp but uses a different fingerprint or distance metric to evaluate the diversity within the set of generated molecules, often providing a more nuanced assessment of internal chemical diversity.

In addition to these metrics, the quality of generated molecules with respect to realistic physical and chemical properties is also evaluated. This has been done by comparing the distributions of properties, including molecular weight (MW), the octanol-water partition coefficient (LogP), Synthetic Accessibility Score (SA), and Quantitative Estimation of Drug-likeness (QED) between the generated and reference molecule sets. These properties are assessed to ensure that the generated molecules not only are structurally plausible but also have properties aligning with those of potentially bioactive compounds.

ChEMBL-large dataset evaluation

We first evaluate generated molecules by SM-RNN, SF-RNN, and self-supervised learning models trained using the ChEMBL-Large dataset, as presented in Table 1. The evaluation results show that these models are effective in generating new valid molecules. Among different input representations, the SELFIES representation significantly outperforms SMILES in terms of molecule validity across all tested models, emphasizing SELFIES representation’s robust encoding of chemically valid structures.

Table 1.

SM-RNN, SF-RNN and Self-supervised learning (SMILES & SELFIES) evaluation result with the ChEMBL-Large dataset

Dataset: ChEMBL	SM-RNN	SF-RNN	SS 5%		SS 20%
Representation	SMILES	SELFIES	SMILES	SELFIES	SMILES	SELFIES
Valid [0,1]↑	0.8945	0.9883	0.798	0.989	0.6728	0.9885
Unique@1000 [0,1]↑	0.983	0.998	0.976	0.993	0.968	0.998
Unique@5000 [0,1]↑	0.943	0.985	0.9362	0.9872	0.9304	0.9908
IntDiv [0,1]↑	0.7738	0.7930	0.7633	0.7948	0.7437	0.7979
IntDiv2 [0,1]↑	0.7532	0.7777	0.7426	0.7797	0.7222	0.7832
Filters [0,1]↑	0.2740	0.1730	0.2838	0.1698	0.2781	0.1612
Novelty [0,1]↑	0.4906	0.9154	0.6943	0.9573	0.7937	0.9765
Average logP	−2.8062	−1.9244	−3.1766	−1.9303	−3.8540	−1.8537
Average SA [1,10]	7.1539	7.2053	7.1869	7.1691	7.2441	7.1493
Average QED [0,1]	0.0317	0.0358	0.0305	0.0368	0.0284	0.0396
Average weight	1811.2404	1714.9326	1823.4457	1705.8983	1864.1485	1676.8556

Metric arrows indicate whether higher (↑) or lower (↓) values are preferred. SELFIES representations notably enhance the validity and uniqueness of generated molecules, underscoring their robustness for valid structure generation.

For the self-supervised learning models, we observed a clear trade-off between novelty and validity when using SMILES representations, particularly noticeable as the masking percentage increased from 5 to 20%. While increased masking enhanced novelty by encouraging the exploration of diverse chemical space, it simultaneously resulted in reduced validity. In contrast, the SELFIES representation maintained high validity regardless of masking percentage, highlighting its resilience. To further characterize this behavior, we have evaluated a broader range of masking ratios during preliminary experiments, including extreme masking levels of up to 90%, to examine model robustness and failure modes. As summarized in Supplementary Table 1, SELFIES-based models preserve high validity even under aggressive masking, whereas SMILES-based models degrade substantially. Based on these observations, we report results using 5–20% masking in the main experiments as a representative regime that clearly illustrates the novelty-validity trade-off while maintaining stable training and high validity.

The internal diversity metrics (IntDiv and IntDiv2) and uniqueness metrics (Unique@1000 and Unique@5000) consistently demonstrate the model’s capacity to generate structurally diverse and distinct molecules. This ability is critical for effectively exploring chemical space for novel discoveries (Fig. 1).

Fig. 1. Flowchart illustrating the comparative analysis of generative AI models for large molecule design.

A Traditional process for small molecule generation, starting from dataset collection, model selection, to generation and design. B Illustration of our work, which emphasizes the use of large molecule datasets such as ChEMBL-Large, highlighting the diversity in molecular sizes. It also details the various generative models tested. C Evaluation and comparison phase, where the performance of the models is assessed using metrics such as validity, novelty, uniqueness, molecular weight, and synthetic accessibility. This comprehensive approach underscores the effectiveness of different models in handling the complexities associated with large molecular structures.

We further compare the properties of generated molecules against the original ChEMBL-Large dataset (average weight 1798.953 Da, logP −2.558, SA 7.106, QED 0.032). A wide variety of evaluation metrics, such as logP (lipophilicity), SA (Synthetic Accessibility), QED (Quantitative Estimate of Drug-likeness), and molecular weight, are used to assess how closely the generated molecules align with desirable properties in the chemical space.

In our visual analysis (Fig. 2 and Table 2), we have observed that the generated molecules share notable structural and chemical similarities with molecules from the ChEMBL-Large dataset. We also provide additional visual analysis for substantially heavier molecules in the Supplementary Fig. 1 and Supplementary Table 3, further supporting the consistency of our generative models across a wider molecular weight range. The close alignment between visual representations and computed property metrics confirms that our models effectively capture and reproduce relevant features from the training data.

Fig. 2. Comparative visualization of molecular structures from the ChEMBL-Large dataset and the generated ones by different models.

a Example molecule from the ChEMBL-Large dataset, b molecule generated by the SM-RNN model, c molecule generated by the Self-Supervised model with SMILES representations, and d molecule generated by the Self-Supervised model with SELFIES representations. All generated molecules have molecular weights around 1300 Da.

Table 2.

Detailed properties of 2D molecular structures

Figure	Source	Weight	logP	SA	QED
2a	Train	1381	−1.91	7.63	0.043
2b	SM-RNN	1388	−1.74	7.01	0.013
2c	SS SM	1347	−1.72	6.95	0.035
2d	SS SF	1264	−2.36	7.15	0.012

The values in the table are for single representative molecules from each source. The results indicate that all of our models trained on the ChEMBL-Large dataset are capable of generating new valid molecules with similar properties as those of the training set, a conclusion supported by both visual comparisons and property metrics.

Complete evaluation results on the MOSES dataset are provided in the Supplementary Table 2.

Mixed dataset evaluation (ChEMBL + MOSES)

We further evaluate our models on a mixed dataset combining ChEMBL-Large and MOSES to assess generalization across heterogeneous molecular size distributions. This setting introduces substantial variability, as the combined dataset includes a large number of small molecules (MOSES, average ~307 Da) alongside substantially larger and more structurally complex molecules (ChEMBL-Large, average ~1799 Da). Such heterogeneity provides a stringent test of whether generative models can learn beyond dominant or simpler molecular patterns.

Across models, we observe trends broadly consistent with single-dataset evaluations (Table 3), with a reduced validity-novelty trade-off due to the presence of smaller molecules in the mixed dataset. To examine whether models preferentially sample particular size regimes, we compare marginal property distributions of generated molecules against the mixed training distribution (Fig. 3). The SM-RNN baseline closely reproduces the training-set marginals for molecular weight and related physicochemical properties, reflecting strong distribution matching but also suggesting a tendency to replicate dominant mixture modes. In contrast, the self-supervised SELFIES model with 20% masking exhibits broader coverage across the mixed distribution, indicating reduced reliance on the most frequent small-molecule patterns. The Transformer with APE achieves high validity and diversity and shows clear evidence of generalization; however, its generated samples concentrate more strongly on a specific subset of the mixed distribution, highlighting that lightweight attention-based models may still exhibit mode preference under heterogeneous size regimes.

Table 3.

SM-RNN, SF-RNN and Self-supervised learning (SMILES & SELFIES) evaluation result with the ChEMBL + MOSES dataset

Dataset: ChEMBL + MOSES	SM-RNN	SF-RNN	SS 5%		SS 20%
Representation	SMILES	SELFIES	SMILES	SELFIES	SMILES	SELFIES
Valid [0,1]↑	0.9375	0.9945	0.8783	0.9934	0.783	0.995
Unique@1000 [0,1]↑	0.998	0.999	0.994	1	0.996	1
Unique@5000 [0,1]↑	0.9818	0.9922	0.9832	0.9952	0.9798	0.9958
IntDiv [0,1]↑	0.8532	0.8613	0.8515	0.8622	0.8486	0.8634
IntDiv2 [0,1]↑	0.8376	0.8481	0.8357	0.8493	0.8315	0.8508
Filters [0,1]↑	0.6518	0.5459	0.6531	0.5337	0.6602	0.5264
Novelty [0,1]↑	0.6682	0.9026	0.8470	0.9652	0.8917	0.9818
Average logP	−0.1423	0.2250	−0.1926	0.1980	−0.4734	0.3216
Average SA [1,10]↓	4.7508	4.9411	4.6832	4.9499	4.6866	4.9775
Average QED [0,1]↑	0.4254	0.4143	0.4373	0.4087	0.4444	0.4073
Average weight	1036.2791	1005.4331	1020.2633	997.4128	1035.2478	976.1220

Fig. 3. Distribution of molecular weight, logP, synthetic accessibility (SA), and QED for generated molecules from different models trained on the mixed ChEMBL + MOSES dataset.

Training dataset distributions are overlaid in each plot.

For the combined dataset, the average training-set properties are molecular weight 1053.044 Da, logP −0.060, SA 4.778, and QED 0.419. Across models, generated molecules broadly reflect these aggregate statistics, indicating that none of the approaches collapse entirely to a single molecular size regime. At the same time, closer inspection of the marginal distributions reveals distinct behaviors: some models closely reproduce the training-set distributions, whereas others exhibit broader coverage or preferential sampling of specific subregions within the mixture. In particular, self-supervised masking appears to mitigate over-reliance on dominant small-molecule patterns, contributing to improved robustness under dataset heterogeneity. Although this analysis is not explicitly stratified by molecular weight, the observed trends motivate more systematic size-conditional or bin-wise evaluations as an important direction for future work.

Mixed dataset evaluation with APE tokenization

Further evaluations employing APE tokenization on the mixed dataset demonstrate robust and consistent generative performance across all baseline architectures, including the decoder-only Transformer, SF-RNN, and self-supervised RNN variants (Table 4). By reducing the average sequence length from approximately 122 tokens under the original encoding to around 50 tokens (Supplementary Table 4), APE substantially improves computational efficiency and memory usage. This compression enables attention-based models, whose self-attention complexity scales quadratically with sequence length, to be trained under the same experimental constraints as recurrent baselines.

Table 4.

Evaluation of Transformer, SF-RNN, and self-supervised models with APE tokenization on the ChEMBL + MOSES dataset

Dataset: ChEMBL + MOSES
Model	Transformer + APE	SF-RNN + APE	SS 5% + APE	SS 20% + APE	SS 60% + APE	SS 80% + APE
Representation	SELFIES	SELFIES	SELFIES	SELFIES	SELFIES	SELFIES
Valid [0,1]↑	0.9991	0.9952	0.9947	0.9960	0.9960	0.9967
Unique@1000 [0,1]↑	0.9990	0.9980	1.0000	0.9990	1.0000	0.9990
Unique@5000 [0,1]↑	0.9986	0.9922	0.9932	0.9962	0.9994	0.9986
IntDiv [0,1]↑	0.8815	0.8651	0.8660	0.8695	0.8756	0.8822
IntDiv2 [0,1]↑	0.8689	0.8523	0.8534	0.8572	0.8634	0.8707
Filters [0,1]↑	0.3190	0.4552	0.4413	0.4088	0.3046	0.2457
Novelty [0,1]↑	0.9978	0.9749	0.9844	0.9907	0.9960	0.9991
Average logP	0.5589	0.2128	0.2721	0.3558	0.4730	0.6711
Average SA [1,10]↓	5.0553	5.1912	5.2290	5.2785	5.4838	5.5532
Average QED [0,1]↑	0.3736	0.3854	0.3772	0.3679	0.3305	0.3059
Average weight	723.68	985.28	979.62	956.16	914.09	835.28

Metric ranges are shown in brackets. Arrows indicate whether higher (↑) or lower (↓) values are preferred.

Despite the aggressive reduction in sequence length, models trained with APE maintain comparable or improved performance in terms of validity, uniqueness, and internal diversity, while consistently achieving higher novelty. Notably, recurrent and self-supervised models with APE perform on par with, and in some metrics exceed, the Transformer baseline, indicating that efficient tokenization and representation learning can yield strong generative performance even with relatively lightweight architectures.

Beyond unconditional evaluation, we conduct a preliminary conditional generation study to examine whether simple conditioning mechanisms can induce directional control over key physicochemical properties in a heterogeneous molecular setting. Rather than introducing a fully optimized conditional architecture, this analysis evaluates the feasibility of property-aware generation using lightweight extensions of existing sequence models.

We employ a prefix-based conditioning scheme in which discretized property tokens are prepended to each SELFIES sequence during training. Molecular weight (MW) and logP are discretized into quantile bins using statistics computed from the training split and encoded as fixed-order conditioning tokens preceding the molecular string. During training, conditioning tokens are randomly masked with a small probability to enable robustness to partially specified conditions. At inference time, molecules are generated under low, medium, and high target settings for individual properties while masking others, allowing isolation of each conditioning variable.

Using this setup, we evaluate conditional generation with the self-supervised SELFIES model with APE and 20% masking, which demonstrated strong generalization performance in prior mixed-dataset experiments. As shown in Table 5 and Fig. 4, generated molecular weight and logP distributions shift systematically in the expected direction under different target settings. Compared to a simpler conditional RNN baseline without self-supervised masking (see Preliminary Conditional Generation Analysis Baseline in the Supplementary Table 9 and Supplementary Fig. 4, conditional control is more pronounced with the self-supervised SELFIES model, particularly for molecular weight. While variance remains substantial—especially for logP—these results demonstrate measurable directional controllability and motivate the exploration of more expressive conditional architectures as future work.

Table 5.

Conditional generation results using the self-supervised SELFIES (20%) model with APE tokenization

Condition	Property	Target	Gen. Mean	Gen. Std
MWlow	Weight	289.34	263.78	38.91
MWmid	Weight	733.11	969.61	250.53
MWhigh	Weight	1995.37	1655.73	533.50
logPlow	logP	0.26	0.47	3.21
logPmid	logP	2.21	1.93	1.92
logPhigh	logP	3.07	2.37	1.77

Reported values show the target property value used for conditioning and the mean and standard deviation of generated molecules.

Fig. 4. Conditional generation results using the self-supervised SELFIES (20%) model with APE tokenization.

Generated distributions for logP (left) and molecular weight (right) shift systematically in response to low, mid, and high conditioning targets, indicating measurable directional control under heterogeneous training conditions.

Summary of experimental findings

Our experiments systematically characterize the strengths and limitations of sequence-based generative modeling approaches for large molecule design. The SELFIES representation consistently maintains high validity, even under aggressive self-supervised masking, demonstrating its suitability for modeling long and complex molecular sequences. In addition, the proposed Atom-Pair Encoding (APE) tokenization strategy provides a substantial computational benefit by significantly reducing effective sequence length while preserving, and in some cases improving, generative performance across multiple evaluation metrics.

Taken together, these results indicate that the combination of efficient tokenization via APE and targeted self-supervised learning constitutes an effective strategy for improving scalability, robustness, and generalization in large-molecule generative models. The observed gains in validity, diversity, novelty, and distributional fidelity establish a strong empirical foundation for future efforts aimed at developing efficient and broadly applicable generative frameworks for complex molecular design.

Discussion

In this work, we evaluated sequence-based generative AI models for large molecule design and systematically examined their scalability, robustness, and generative behavior. Among the explored approaches, autoregressive molecular language models—including RNN- and LSTM-based architectures—demonstrate strong potential for scaling to complex molecular datasets. In particular, models employing SELFIES representations consistently achieve high validity, even under substantial self-supervised masking, highlighting the robustness of grammar-constrained string representations for large-molecule generation.

To further improve efficiency and scalability, we introduced Atom-Pair Encoding (APE) as a tokenization strategy for long molecular sequences. APE substantially reduces sequence length, leading to improved computational efficiency while achieving generative performance that is comparable to or better than baseline models across multiple evaluation metrics. These results demonstrate that careful tokenization design, combined with self-supervised learning, can meaningfully enhance both model efficiency and generalization without sacrificing chemical realism.

We further analyzed the distributions of generated molecular properties, including molecular weight, logP, synthetic accessibility (SA), and quantitative estimate of drug-likeness (QED), and found that the proposed models reliably reproduce key characteristics of the training data. Evaluations on a heterogeneous mixed dataset highlight improved generalization behavior, particularly for models incorporating APE and self-supervised masking, which mitigate over-reliance on dominant molecular subpopulations.

Looking forward, continued refinement of molecular language model architectures, training strategies, and tokenization schemes will be important for further advancing large-molecule generative modeling. Extensions of self-supervised learning, including adaptive or structure-aware masking strategies, represent promising directions for improving controllability and coverage of chemical space, although such developments are beyond the scope of the present study.

Beyond small- and medium-sized molecule generation, sequence-based generative modeling is increasingly relevant for protein-ligand and biologics design, including peptides, macrocycles, and antibody-derived molecules, where candidates are often large, flexible, and challenging to model using structure-centric pipelines. Recent progress in these areas has demonstrated the effectiveness of sequence-only or weakly structured representations, especially when combined with self-supervised learning or conditional generation strategies, as shown in deep generative models for peptide design¹⁵, protein sequence generation frameworks¹⁶, and large-scale protein language models such as ESM-2¹⁷. In this broader context, our results establish that scalable sequence-based generative models can reliably produce chemically valid and diverse large molecules, providing a practical foundation for extending generative modeling toward biologically relevant applications. While such conditional and biologically informed generation tasks are beyond the scope of the present work, the findings here position sequence-based models as a flexible and extensible framework for large-molecule and biologics-oriented drug discovery.

Overall, this study takes an important step toward improving the efficiency and scalability of generative models for large molecule design. By demonstrating that careful representation choices, efficient tokenization, and self-supervised learning can jointly overcome key limitations of existing approaches—challenges frequently highlighted in recent chemoinformatics reviews¹⁸—our work contributes to ongoing efforts to enable data-driven molecular design at scales relevant to modern drug discovery and materials science.

Methods

We evaluate generative molecular models with a focus on generating large molecules, a task that introduces challenges in model scalability, chemical validity, and computational efficiency. While several existing models have demonstrated success on small molecule datasets, their effectiveness often diminishes as molecular size increases. Graph-based models utilizing normalizing flows or variational autoencoders (VAEs) proved computationally prohibitive for large molecules due to memory and runtime constraints. Thus, we restrict our evaluation to models and tokenization strategies that successfully scale to larger molecular structures. In particular, we benchmark existing models and introduce self-supervised generative learning as a novel enhancement for robust and valid large molecule design. Our findings inform future directions in efficient generative AI for materials science and drug discovery.

Baseline models

Recurrent neural networks (RNN) & long short-term memory (LSTM)

Recurrent neural networks (RNN) & long short-term memory (LSTM). Our generative models primarily employ RNNs¹⁹, particularly the LSTM²⁰ variant, well-known for handling long-range dependencies and mitigating vanishing gradients. LSTM updates character embeddings using gates (input i_t, forget f_t, cell candidate g_t, and output o_t), hidden states (h_t), and cell states (c_t) as:

$$\begin{array}{cc}{i}_t& \begin{array}{ll}=\sigma (W_{ii}{C}_t+b_{ii}+W_{hi}{h}_{t-1}+b_{hi})& \hfill \end{array}\hfill \\ f_t& =\sigma (W_{if}{C}_t+b_{if}+W_{hf}{h}_{t-1}+b_{hf})\hfill \\ g_t& =\tanh (W_{ig}{C}_t+b_{ig}+W_{hg}{h}_{t-1}+b_{hg})\hfill \\ o_t& =\sigma (W_{io}{C}_t+b_{io}+W_{ho}{h}_{t-1}+b_{ho})\hfill \\ c_t& =f_t\odot c_{t-1}+i_t\odot g_t\hfill \\ h_t& =o_t\odot \tanh \left(c_t\right)\hfill \end{array}$$ 1

In these equations, σ denotes the sigmoid function and ⊙ denotes element-wise multiplication.

LSTM predicts the subsequent token in molecular sequences, effectively capturing intricate structural patterns crucial for large molecule datasets.

CharRNN (SM-RNN)

The CharRNN model, explored in ref. ⁵, utilizes recurrent neural networks (RNNs) to generate novel molecules. For clarification, the model referred to as CharRNN will be renamed to SM-RNN here to indicate that it is based on SMILES string representations. Although CharRNN was the initial name given by the original authors, this adjustment will ensure consistency and clarity throughout the manuscript. Given the one-hot embedding of each of the characters C_t, and an initial hidden state h₀ (all zeros), we update each character’s embedding by generating a hidden state h_t as described in the LSTM architecture section.

Given h_t, the character at position t + 1 is predicted as:

$$Pr\left(c_t=v\right)=f_v\left(h_{t-1}\right)\forall v\in \mathcal{V},$$ 2

where $f_v:{\mathcal{R}}^{\mid h\mid }\to \left[0,1\right]$, and ${\sum }_{v\in \mathcal{V}}{f}_v\left(h_{t-1}\right)=1$. An MLP layer followed by a softmax layer is used to implement all f_v()’s. The cross-entropy loss is employed for model training.

To prepare the SM-RNN for large molecule generation, we performed preprocessing steps including normalization of the input strings and adjustment of the model’s batch size and learning rate to accommodate the larger datasets. The RNN, trained on a dataset of known molecules and their properties, successfully learns the underlying structure of these molecules. Notably, the generated molecules correlate well with the training data properties. The model can be fine-tuned using small sets of molecules known to be active against specific targets, enhancing its capability to generate molecules likely to exhibit activity against these targets. This was demonstrated through successful molecule reproduction in tests against biological targets like Staphylococcus aureus and Plasmodium falciparum⁵. The SM-RNN’s ability to complete the entire de novo drug design cycle, generating novel molecules for drug discovery, underscores its effectiveness.

SF-RNN

SF-RNN, discussed in ref. ⁶, is a chemical language model employing a recurrent neural network with LSTM. It is trained on SMILES (SM-RNN) or SELFIES (SF-RNN) representations. The paper highlights the capacity of simple language models like SF-RNN to learn complex molecular distributions effectively. These models, particularly in low-data regimes, have demonstrated better performance than more sophisticated graph generative models. The SF-RNN shows a significant ability to generate valid molecules and is particularly noted for its reduced propensity to overfit, which is a common issue with SMILES language models.

For SF-RNN, additional preprocessing steps were necessary to convert the input dataset to SELFIES format. This required adapting the input pipeline to handle the SELFIES representation and ensuring compatibility with the existing evaluation tools, which primarily accept SMILES. Therefore, generated molecules were converted back to SMILES for evaluation. Despite these additional steps, SF-RNN maintained high validity and uniqueness metrics, highlighting its robustness and adaptability.

Transformer (decoder-only)

Transformers²¹ replace recurrent updates with self-attention mechanisms, enabling direct modeling of long-range dependencies through multi-head attention modules. We also implement a lightweight decoder-only Transformer baseline to compare against RNN/LSTM-based models under identical data preprocessing and training protocols. For autoregressive sequence generation, our decoder-only transformer applies a causal (triangular) attention mask so that each token attends only to preceding positions, forming a natural language-model-style generator for molecular strings. The model embeds SELFIES tokens, incorporates sinusoidal positional encodings, and processes sequences through a small stack of transformer layers, with output logits trained via next-token prediction using cross-entropy loss. Autoregressive sampling is performed using the same causal attention mask.

Although attention-based models offer improved capacity for modeling long-range dependencies, the quadratic memory and computational scaling of self-attention with respect to sequence length makes training on uncompressed SELFIES sequences impractical under limited available computational resources. This observation motivates the use of sequence compression strategies, such as Atom-Pair Encoding (APE), to enable efficient Transformer-based modeling for large-molecule generation.

Graph-based generative models

Graph-based generative models, including Junction Tree Variational AutoEncoder (JT-VAE)², Graph Auto-regressive Flow (GraphAF)²², Graph Convolutional Policy Network (GCPN)²³, and Molecular Graph RNN (MolRNN)³, among many others, explicitly represent molecules as 2D/3D graphs and can encode rich structural and relational information for molecular design.

To date, however, these models have predominantly been evaluated on small or medium-sized molecular datasets. For example, molecules in the ZINC dataset¹⁴ contain an average of 21 atoms, and those in MOSES¹² contain approximately 23 atoms on average². In contrast, the large-molecule datasets considered in this study contain molecules with an average of 107 atoms, substantially increasing both computational and memory requirements for these graph-based models.

Formally, let $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$ denote a molecular graph, where $\mathcal{V}$ represents atoms or motifs and $\mathcal{E}$ encodes bonding or geometric relationships. Most graph-based generative models adopt an iterative decoding process, in which nodes or edges are generated sequentially, conditioned on the current partial graph ${\mathcal{G}}_t=\left({\mathcal{V}}_t,{\mathcal{E}}_t\right)$. Encoding ${\mathcal{G}}_t$ using graph neural networks (GNNs) typically incurs a computational cost of $\mathcal{O}\left(\mid {\mathcal{V}}_t\mid \cdot d\cdot h^2\right)$, where d denotes the average node degree and h is the latent feature dimension. Consequently, generating a full graph of $\mid \mathcal{V}\mid$ nodes during training leads to an overall complexity of $\mathcal{O}\left({\sum }_{t=1}^{\mid \mathcal{V}\mid }\mid {\mathcal{V}}_t\mid \cdot d\cdot h^2\right)$, which scales quadratically with molecular size.

In our experiments, representative graph-based approaches, including JT-VAE, could not be trained reliably on the large-molecule datasets using available hardware resources. Training consistently terminated due to memory exhaustion prior to convergence, reflecting the unfavorable scaling of graph-based decoding with molecular size. These results highlight the practical limitations of applying existing graph-based generative models to large-molecule generation under realistic computational constraints.

While graph-based and hybrid representations may better capture three-dimensional conformational complexity and long-range interactions in principle, their computational overhead currently limits their applicability to large-molecule datasets at scale. In contrast, our experimental results demonstrate that one-dimensional string-based representations, when combined with self-supervised learning and appropriate tokenization strategies, achieve strong generative performance across validity, novelty, diversity, and property distribution fidelity.

Within the scope of available benchmarks and evaluations, the proposed sequence-based models perform competitively relative to graph-based and hybrid approaches that have primarily been demonstrated on smaller or medium-sized molecules. Accordingly, we adopt SMILES and SELFIES representations as scalable and effective baselines for large-molecule generation, prioritizing practical feasibility while maintaining strong empirical performance. Nonetheless, the research on developing more scalable and efficient graph-based molecular generative models is one of important future research directions.

Self-supervised learning

We introduce self-supervised generative learning as a novel enhancement for improving robustness, scalability, and validity in large molecule design.

To initiate the self-supervised learning process, a critical step involves the strategic masking of characters within the sequential representations of molecules. This is accomplished through a procedure designed to randomly select a subset of characters from the input sequence for masking, based on a predefined masking ratio. Specifically, for each molecule represented by its sequential representation, we calculate the total number of characters to be masked by applying the masking ratio (typically set at 5% and 20% for our experiments). A minimum of one character is masked to ensure the model learns from even the smallest sequences. These characters are then replaced with a special ‘<MASK>’ token in the input sequence, while the original characters are preserved in the target sequence for the model to predict. This method of selective masking and prediction enables our LSTM-based model to effectively learn the intricate patterns and latent structural features inherent in large molecular datasets.

When training with large molecular datasets, our own model incorporates self-supervised learning using LSTM networks to achieve stable training. The LSTM model, with 3 layers and 256 hidden dimensions, along with 128 embedding dimensions, is tasked to predict masked portions of molecular string sequences. This technique enables the model to learn the latent space of the sequence distribution, a crucial aspect for generating complex molecules. The choice of LSTM is motivated by the demonstrated success of the SM-RNN model in handling large molecule datasets, suggesting its suitability as a starting point for self-supervised learning endeavors.

Molecular encoding and tokenization strategies

Basic encoding

We represent molecules as sequential strings using SMILES⁹ or SELFIES¹⁰. SMILES and SELFIES are widely regarded as effective representations for several reasons. SMILES is compact and human-readable, making it easier for chemists to interpret and modify chemical structures. Its linear format simplifies the process of encoding molecules for machine learning models. SELFIES, on the other hand, guarantees 100% validity of generated strings, which addresses a significant limitation of SMILES where invalid strings can often be produced. This robustness makes SELFIES particularly useful in ensuring that generated molecules conform to chemical rules, thus facilitating more reliable downstream applications. Due to these advantages, our study employs both SMILES and SELFIES to leverage their respective strengths in molecular representation. For a molecule m_j, the encoding is:

$$m_j=\left\{c_1,c_2,\dots ,c_n\right\},$$ 3

where each token c_t is an element of the vocabulary $\mathcal{V}$. In SMILES, tokens correspond to atoms or bonds (e.g., ‘C’, ‘=’); in SELFIES, each token like ‘[C]’ or ‘[=C]’ is treated as a single unit-brackets are not separated.

Each token c_t is encoded as a one-hot vector:

$$C_t=\mathrm{OneHot}\left(c_t\right),\mathrm{OneHot\; :}\mathcal{V}\to {\left\{0,1\right\}}^{\mid \mathcal{V}\mid }.$$ 4

This standard encoding forms the baseline for input to our generative models.

Atom-Pair Encoding (APE) tokenization

To address the challenges posed by long sequences and computational overhead in large molecule modeling, we implemented Atom-Pair Encoding (APE), a frequency-based tokenization scheme inspired by Byte Pair Encoding (BPE)²⁴ and adapted from Leon et al.¹¹. APE statistically merges frequently co-occurring adjacent tokens to reduce sequence length while preserving the original string representation.

Starting from basic encoding tokens:

$$m_j=\left\{a_1,a_2,\dots ,a_n\right\},a_t\in \mathcal{V},$$ 5

we iteratively merge the most frequent adjacent pair (a_i, a_i+1) that exceeds a frequency threshold, creating a new token a_i,i+1. After k such merges, we obtain:

$${m_j}^{\prime }=\left\{a_1^{\prime },a_2^{\prime },\dots ,a_m^{\prime }\right\},a_t^{\prime }\in {\mathcal{V}}_{APE},$$ 6

where ${\mathcal{V}}_{APE}$ is the learned APE vocabulary and m≤n.

This tokenization process substantially reduces sequence length (e.g., from approximately 122 to 50 tokens in average in our datasets), improving memory efficiency and computational scalability. Quantitative comparisons of training time, memory usage, and inference efficiency between baseline and APE-based models are reported in the Supplementary Information (see Chemical Interpretability Analysis of Atom-Pair Encoding (APE), Supplementary Table 8). Because APE constructs composite tokens via simple concatenation, the original SELFIES string can be reconstructed exactly by concatenating APE tokens in sequence. APE therefore operates as a lossless compression mechanism at the string level.

APE is driven by token co-occurrence statistics and does not explicitly encode chemical rules. To examine its practical behavior, we analyze reconstruction fidelity, sequence compression characteristics, validity and realism metrics, and representative merged-token examples (see Supplementary InformationChemical Interpretability Analysis of Atom-Pair Encoding (APE), Supplementary Tables 4–7, Supplementary Figs. 2 and 3). These analyses show that APE preserves molecular validity, fragment- and scaffold-level similarity, and recurring local substructures in generated molecules, despite its purely statistical construction.

In summary, APE provides an effective and scalable tokenization strategy for large-molecule sequence modeling, enabling substantial sequence compression while maintaining empirical chemical realism. We therefore adopt APE as a practical balance between representational efficiency and interpretability in the large-molecule regime.

Through these methodologies, our project aims to explore the boundaries of molecular generative modeling, focusing on the scalability and complexity of generating large molecules for applications in drug discovery and materials science.

Ethics statement

This study did not involve human participants, animal subjects, or identifiable personal data. All experiments were conducted using publicly available molecular datasets, and therefore ethical approval and informed consent were not required.

Author contributions

Doyoung Kwak, Xiaoning Qian, and Md Raiyan Chowdhury developed the methodology. Doyoung Kwak and Md Raiyan Chowdhury performed the simulations. Doyoung Kwak and Xiaoning Qian analyzed and interpreted the results. Xiaoning Qian supervised the research. Doyoung Kwak prepared the figures, tables, and initial draft. Xiaoning Qian, Md Raiyan Chowdhury, and Byung-Jun Yoon contributed to manuscript revision. All authors reviewed and approved the final manuscript.

Data availability

All datasets used in this study are publicly available. The ChEMBL-Large dataset was derived from the ChEMBL database (https://www.ebi.ac.uk/chembl), and the MOSES dataset was obtained from the Molecular Sets (MOSES) benchmarking platform (https://github.com/molecularsets/moses), which is based on the ZINC database. Processed data generated during the current study are available from the corresponding author on reasonable request.

Code availability

The code used to train and evaluate the models in this study is available at https://github.com/12doyoung/MolGen_npj.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s44386-026-00052-7.