Generative artificial intelligence based models optimization towards molecule design enhancement

Abstract

Generative artificial intelligence (GenAI) models have emerged as a transformative tool for addressing the complex challenges of drug discovery, enabling the design of structurally diverse, chemically valid, and functionally relevant molecules. Despite significant advancements, the rapid expansion of GenAI applications still faces challenges related to prediction accuracy, molecular validity, and optimization for drug-like properties. This review provides a comprehensive analysis of recent techniques and strategies aimed at enhancing the performance of GenAI models in molecular design. We explore key generative architectures, including variational autoencoders, generative adversarial networks, and transformer-based models, highlighting their unique contributions to drug discovery. Additionally, we discuss critical advancements such as reinforcement learning, multi-objective optimization, and the integration of domain-specific chemical knowledge, which collectively enhance molecular validity, novelty, and drug-likeness. Also, the review examines persistent challenges, including data quality limitations, model interpretability, and the need for improved objective functions, while offering insights into future research directions. By mapping the evolving landscape of GenAI-driven molecular design and providing strategic guidance for overcoming existing limitations, this review serves as an essential resource for researchers leveraging GenAI in drug discovery.

Introduction

Drug discovery is a complex, multi-stage process that requires substantial time and resources [1]. Identifying potential drug targets in preclinical studies is challenging due to the inherent biological complexity and insufficient target validation [2]. However, advancements in computing power, coupled with sophisticated algorithms and high-throughput screening techniques, offer tremendous potential to accelerate drug discovery. These innovations enhance various aspects of the process, including target identification, hit compound screening, and molecular design optimization [3]. Generative Artificial Intelligence (GenAI) is revolutionizing molecular design by providing advanced tools for generating novel molecular structures tailored to specific functional properties [4]. Traditional molecular generation methods, which rely on combinatorial synthesis and optimization, have long been constrained by computational and experimental limitations. However, the emergence of generative models—particularly those driven by machine learning (ML) architectures such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and diffusion models—has transformed the field, allowing researchers to explore vast chemical spaces with unprecedented depth and efficiency [5, 6]. By leveraging sophisticated molecular interaction modeling and property prediction, GenAI streamlines the discovery process, unlocking new possibilities in drug development, materials science, and chemical engineering. Table 1 provides a glossary of key technical terms utilized throughout this paper.

Table 1.

Definitions of technical terms covered in this paper (in alphabetical order)

Term	Definition
Active learning	A machine learning approach where the model is trained iteratively by selecting the most informative data points (e.g., molecules) for labeling, thus improving efficiency in discovering optimal molecules [18]
Autoregressive predictions	A predictive model where the output depends on its previous states or outputs. Commonly used in sequence generation, including molecular generation in SMILES format [19]
Combinatorial chemistry	The rapid synthesis or computational generation of large libraries of structurally diverse molecules for screening purposes [20]
Curriculum learning	A training strategy where the model is initially presented with simpler and progressively more complex tasks, improving learning stability [21]
Diffusion model	It is a generative model that learns to gradually denoise a signal, reversing a forward process that progressively adds noise to data [22]
DRD2	Dopamine receptor D2, are often used as a target in molecular design for drugs treating neurological disorders [23]
Exploitation	A strategy in optimization where the model focuses on exploring known promising regions in the solution space [24]
Exploration	Unlike exploitation, it involves investigating unknown regions of the solution space to find novel solutions [24]
Fréchet chemNet distance	A metric used to evaluate the similarity between the distributions of two sets of molecular representations [25]
Functional groups	Specific groups of atoms within molecules that determine their chemical reactivity
Generative adversarial learning	A framework involving two networks (generator and discriminator) competing to improve the quality of generated samples, such as molecules [26]
Global fitness metric	A quantitative measure used to evaluate the overall quality or effectiveness of a generated molecule, often considering multiple properties like binding affinity, drug-likeness, etc. [27]
Heuristic optimization	Techniques that use experience-based strategies for problem-solving, often employed to optimize molecular properties
Information entropy maximization	A strategy in AI to enhance diversity in molecular generation by maximizing the entropy of molecular property distributions
Inverse molecular design	A computational approach that aims to generate molecular structures with specific desired properties by reversing the typical structure–property relationship [28]
Language model	A neural network model trained to predict sequences of tokens, such as SMILES strings in a molecular generation [17]
Latent space	A continuous vector space representation where complex data (e.g., molecular structures) is encoded for learning and optimization
Library design	The creation of a diverse and representative set of molecules for use in screening or optimization workflows [29]
Linker design	The design of molecular components (linkers) connecting two or more active fragments, crucial in drug discovery [30]
LogP	A measure of a molecule’s hydrophobicity or lipophilicity, representing the partition coefficient between octanol and water
Markov chain	A stochastic process where the next state depends only on the current state, used in probabilistic modeling for the molecular generation
Mode collapse	A failure mode in generative adversarial networks (GANs) where the generator produces limited diversity, often a problem in molecular generation [10]
Molecular fragments	Smaller substructures of molecules that can be recombined to form new compounds in molecular design
Parallel tempering	A Monte Carlo technique that uses multiple simulations at different temperatures to overcome local minima are often used in molecular simulations [31]
Policy gradient algorithm	A reinforcement learning algorithm optimizing policies by maximizing expected rewards, applied in molecular property optimization
QED	A measure quantifying how drug-like a molecule is based on properties like molecular weight, lipophilicity, and hydrogen bond donors/acceptors [32]
RealFormer	An optimized transformer architecture that uses simplified attention mechanisms for computational efficiency in generative models [33]
Reinforcement learning	A machine learning approach where agents learn to make decisions by maximizing cumulative rewards are widely used in molecular design [24]
RNN	A type of neural network designed for sequential data, used in modeling molecular SMILES strings
SAscore	A measure of the synthetic feasibility of a molecule, balancing complexity and potential synthetic challenges [34]
Scaffold hopping	The process of discovering structurally novel molecules with similar biological activity by modifying core molecular structures [35]
SELFIES	A robust, grammar-aware molecular string representation designed to overcome syntactical errors in generative models [36]
SMILES	A textual format for representing chemical structures as sequences of characters
Stochastic context learning	Learning techniques that incorporate randomness to improve generalization in uncertain or noisy environments, applicable to molecular generation [37]
Syntactical errors	Errors in the molecular representation (e.g., invalid SMILES strings)
transfer learning	A technique where a pre-trained model on one task is fine-tuned for a different but related task, widely used in molecular property prediction [38]
Transformer	A deep learning architecture based on attention mechanisms, effective for sequence modeling tasks, including SMILES generation [19]
Variational auto encoder	A generative model that learns latent representations by encoding and decoding probabilistic distributions [39]

Each type of generative model has its own unique working principles and characteristics, making each of them suitable for different generative tasks and fields of application [7]. For example, Variational Autoencoders (VAEs) are generative neural networks that encode input data into a lower-dimensional latent representation and then reconstruct it from sampled points. This approach ensures smooth latent space, enabling realistic data generation. Variants such as Deep VAEs, InfoVAEs, and GraphVAEs are particularly valuable in bioinformatics, materials science, graph generation, and molecular design [8]. Transformers, originally developed for natural language processing (NLP), are deep learning models designed for tasks such as machine translation, text summarization, and question-answering.

Transformers are used to efficiently process the data with long dependencies, with reduced training time based on a parallelizable architecture. The architecture of the parallelizable architecture includes an encoder-decoder structure incorporated with self-attention layers, positional encoding, and multi-head attention [9]. This combination of features makes Transformers suitable for learning subtle dependencies in data, which in turn is particularly useful for handling complex tasks while enhancing computational efficiency and adaptability for applications like face editing and image analysis. GANs, however, rely on two independent and competing networks, a generator for creating synthetic data, and a discriminator for distinguishing real from generated data, both operating in an iterative training manner [10]. This iterative adversarial process allows the use of GANs in critical applications like image synthesis, context creation, creative content generation, and domain translation. As for the diffusion models, they take a different approach by progressively generating noise in a clean data sample and learning how to reverse this process by denoising it [11]. This process is based on probabilistic modeling of capturing complex data distributions. Denoising Diffusion Probabilistic Models (DDPMs) and Score-based Generative Models (SGMs) have demonstrated great potential and exceptional performance in performing high-quality image synthesis and generative modeling tasks.

The research trend of applying GenAI to molecular design leverages its unique capacity to produce de novo structures with specific attributes and properties. Recent advancements include text-guided and diffusion-based models that allow for certain targeted generations of small molecules suitable for drug development. For instance, some works demonstrated a diffusion-based framework that integrates AI-driven capabilities for refining molecular structures to meet desirable biochemical properties [5]. Such efforts address the challenges of traditional time-intensive synthesis methods and meet critical needs in pharmaceuticals, where finding optimal molecular configurations can be prohibitively time and recourse-intensive. Beyond small molecules, AI-driven methods have also been used to design proteins and polymers while creating optimized structures for interaction with biological targets or physical performance in materials [12]. These developments highlight the expanding role of AI in designing highly complex molecular forms, including macromolecules and biopolymers, as well as integrating AI into computational chemistry pipelines to enhance predictive accuracy.

The overarching aim of employing GenAI in molecular structure generation is twofold: (1) to accelerate the discovery of novel molecules with specified functional properties, and (2) to enhance the efficiency and efficacy of computational design methods by reducing trial-and-error processes. As the field advances, AI-driven molecular generation increasingly facilitates "goal-directed" molecule synthesis, where specific therapeutic or material properties are directly encoded into the generative process [13]. This approach speeds up the discovery of high-potential compounds, contributes to cost efficiency, and minimizes the experimental testing required.

Our goal is to highlight the growing impact of GenAI in molecular design and showcase its broad application in creating automated pipelines for molecular generation across diverse applications. Unlike the review introduced by Xia et al. [14], which summarizes the GenAI applications in drug discovery, this review presents the technical key optimization strategies used for molecular design, including property-guided generation, reinforcement learning frameworks, Bayesian optimization techniques, and multi-objective optimization approaches. Building on the foundation of these optimization techniques, we discuss the integration of these techniques with GenAI models, emphasizing how their combined use enhances the generation of novel and functionally relevant molecular structures. The paper further addresses existing critical challenges and limitations in applying GenAI to molecular design, such as the complex chemical space, inaccuracy of results, data scarcity, model interpretability, and computational scalability. Finally, emerging trends and future directions are highlighted, focusing on the potential advancements that could further enhance AI-driven molecular design and optimization processes.

Optimization strategies

Generating chemically valid and functionally relevant molecules is challenging due to the complex chemical spaces and possibilities of structures. Optimization strategies address this challenge, facilitate molecular generation, and guide generative models toward specific target properties. These strategies refine the molecular generation process, improve the model’s performance, efficiency, and accuracy, and enhance the overall quality of predicted molecular structures. Integrating optimization strategies into the models can produce novel, valid, and unique chemical structures. The molecular generation process is guided by these strategies by coordinating the output of models with specific design conditions, such as improving properties, binding affinity, chemical stability, etc. Optimization techniques enable models to learn from past iterations and adjust their generative process accordingly by incorporating feedback mechanisms, thus improving the likelihood of discovering valid and novel molecules. In this paper, we provide an overview of the optimization strategies used for molecule design.

Property-guided generation

Property-guided generation advances the field of molecular design, offering a guided approach to generating molecules with desirable objectives. For instance, the Guided Diffusion for Inverse Molecular Design (GaUDI) framework, introduced by Weiss et al. [15] combines an equivariant graph neural network for property prediction with a generative diffusion model. This approach demonstrated significant efficacy in designing molecules for organic electronic applications, achieving validity of 100% in generated structures while optimizing for both single and multiple objectives [16]. Another innovative approach in this domain is the deployment of VAEs for property-guided generation. Sanchez and Guzik [17] explored the application of VAEs in inverse molecular design, emphasizing their ability to efficiently direct the vast chemical space.

The integration of property prediction into the latent representation of VAEs allows for a more targeted exploration of molecular structures with properties.

Reinforcement learning approaches

Reinforcement learning (RL) has emerged as an effective tool in molecular design optimization, which involves training an agent to navigate through molecular structures. In this context, reward function shaping is crucial for guiding RL agents toward desirable chemical properties such as drug-likeness, binding affinity, and synthetic accessibility. Models like MolDQN modify molecules iteratively using rewards that integrate these properties, sometimes incorporating penalties to preserve similarity to a reference structure [40, 41]. However, a key challenge in the RL setting is balancing exploration and exploitation, in which agents must search for new chemical spaces for diversity while refining known high-reward regions. Bayesian neural networks help manage uncertainty in action selection, while techniques like randomized value functions and robust loss functions further enhance this balance [42]. For example, You et al. developed the graph convolutional policy network (GCPN) that uses RL to sequentially add atoms and bonds, constructing novel molecules with targeted properties [43]. The GCPN demonstrated remarkable results in generating molecules with desired chemical properties while ensuring high chemical validity. Furthermore, Khemchandani et al. [12] designed DeepGraphMolGen, employing a graph convolution policy and a multi-objective reward to generate molecules with strong binding affinity to dopamine transporters while minimizing binding to norepinephrine receptors, illustrating the effectiveness of RL in complex molecular optimization tasks. In another study, Shi et al. [16] developed GraphAF, an autoregressive flow-based model that integrates the strengths of flow-based generative models with RL fine-tuning. This hybrid approach provides efficient sampling from the latent learned distribution and targeted optimization towards desired molecular properties.

Bayesian optimization

In molecular design, Bayesian optimization (BO) is used particularly when dealing with expensive-to-evaluate objective functions. This approach develops a probabilistic model of objective function, providing informed decisions about the evaluation of the next candidate. BO is well-suited for problems where the evaluation of candidate molecules is costly, such as docking simulations or quantum chemical calculations. BO navigates high-dimensional chemical or latent spaces to identify molecules with optimal properties. In generative models, BO often operates in the latent space of architectures like VAEs, proposing latent vectors that are likely to decode into desirable molecular structures [44]. For example, Gómez-Bombarelli et al. [45] integrated Bayesian optimization with deep learning techniques. They utilized a VAE to learn a continuous representation of molecules and perform Bayesian optimization in this learned latent space, leading to a more efficient exploration of chemical space. However, integrating BO into latent spaces presents challenges due to the complex and often non-smooth mapping between latent vectors and molecular properties. Effective kernel design is essential—techniques such as projecting policy-invariant reward functions to single latent points can enhance exploration, as seen in BO-IRL [46]. Additionally, acquisition functions must carefully balance exploration of uncertain regions with exploitation of known optima. One promising approach is multi-step lookahead BO, which plans several moves ahead in latent space, often in combination with reinforcement learning, and has shown improved sample efficiency over standard greedy BO in molecular benchmark tasks [47]. Tripp and Hernández-Lobato [41] highlight several recurring issues in this context, including poor kernel selection, overconfident uncertainty estimates, and unreliable optimization of acquisition functions. Their work offers diagnostic tools and practical fixes, such as input warping, adaptive acquisition noise, and surrogate calibration, that improve BO reliability in high-dimensional latent spaces. Recent work by Korovina et al. [48] created a novel Bayesian optimization algorithm, called ChemoBO, specifically designed for molecular optimization, allowing BO to operate in a domain-aligned feature space. Moreover, it integrates synthesizability constraints to ensure that recommended molecules are chemically valid, solving the BO-related issue, which involves handling discrete molecular spaces and ensuring synthetic feasibility.

Multi-objective optimization techniques

As real-world applications often require balancing multiple properties, the application of multi-objective optimization in molecular design is crucial. This approach aims to provide optimal solutions that represent the best trade-offs between different objectives. In the context of the molecular generation approach, drug discovery often involves multiple conflicting objectives, such as maximizing potency while minimizing toxicity or improving selectivity while maintaining solubility. These can be implemented via scalarization, where multiple objectives are weighted into a single reward, or via Pareto optimization, which seeks non-dominated solutions across objectives [14]. Notably, RL often uses scalar reward functions that combine properties like logP, QED, and synthetic accessibility into a single objective; however, this approach can obscure trade-offs between competing goals. Models like NSGA-II-enhanced molecular generators maintain a Pareto front of solutions that trade off different objectives, giving users more flexibility in candidate selection [49]. This strategy is especially useful in latent generative models and is effectively implemented in DeepGraphMolGen to optimize binding selectivity across biological targets, as mentioned before. For further improvement of RL agent’s exploration, Blaschke et al. [50] introduced memory-assisted RL, which provides diversity-driven replay buffers, encouraging agents to explore novel chemical regions without sacrificing performance.

Integration of optimization techniques into generative models

AI models have exhibited outstanding abilities in exploring enormous chemical spaces and generating molecules with targeted properties [51]. Using inverse design, these models leverage sophisticated ML techniques to predict molecular properties and develop new molecular structures that meet specific criteria. This section reviews generative architectures commonly used in molecular design, focusing on models based on VAEs (e.g. JT-VAE, MolVAE, and MOLAR), GANs (e.g. MolGAN and EarlGAN), and Transformer-based autoregressive frameworks (e.g. REINVENT, RM-GPT, GenSMILES, and Llamol). These approaches have formed the backbone of early molecular generative modeling efforts, often integrated with optimization techniques such as reinforcement learning and Bayesian search. This enhances the validity, drug-likeness, and functional relevance of generated molecules [52]. In the following section, diffusion models are discussed, which have recently proven to be a powerful alternative, particularly for 3D structure-conditioned generation. Fig. 1 demonstrates the whole process of the integration of optimization techniques with the GenAI models to optimize the molecule design generation.

Fig. 1.

Overall Overview of how the GenAI model works for molecule design optimization. The process includes data collection (e.g. Drugs, Proteins, DNA, Polymers, etc.) and preparation to convert them into a suitable format (e.g. SMILES, SELFIES, and Descriptors). Then, the GenAI model selection includes transformers, GAN, VAE, RNN, and Diffusion. To evaluate the model's performance, some metrics need to be calculated, including validity, novelty, uniqueness, and diversity. Based on that, the optimization strategies are deployed to fine-tune the model to achieve a high validity score

Transformer-based models

REINVENT 4

REINVENT 4 is an advanced GenAI framework that optimizes molecular design through reinforcement learning and deep learning techniques. Primarily supported by the Molecular AI team at AstraZeneca, REINVENT aims to enable the de novo generation of molecular structures that meet pre-specified properties relevant to drug discovery and material science. Loeffler et al. [53] designed an open-source AI framework for small molecule design called REINVENT 4, employing recurrent neural networks and transformer architectures within a comprehensive machine learning environment that includes optimization, transfer learning, reinforcement learning, and curriculum learning. It supports applications such as R-group modification, de novo synthesis, linker design, library design, molecule optimization, and scaffold hopping. This framework is configured via TOML or JSON, aiming to provide reference implementations for key AI-based molecular generation algorithms and to serve as a platform for education and innovation in molecular design. REINVENT 4 has introduced various innovations to enhance molecular design and it consists of 4 generators. First, REINVENT which consists of RNN which sequentially creates new molecules atom by atom aiming for de novo design. Libinvent enables the R-replacement and library design which is essential for scaffold decoration. Then, Linkinvent identifies a linker between two fragments, and it enables scaffold hopping, which involves altering the core structure of a molecule while maintaining its functional properties. Finally, the Mol2Mol generator serves as molecular optimization, where the generator provides an alternative molecule within a conditioned similarity.

REINVENT 4’s impact extends across multiple domains. In pharmaceutical research, it aids in generating drug candidates with optimized ADME properties, while in material science, it enables the creation of molecules with specific physical properties for applications like polymer synthesis and catalyst design. In another approach, Sha and Zhu [54] proposed a molecular generation method that addresses the challenge of producing drug molecules without targeted pharmacological properties. The approach leverages reinforcement learning through the policy gradient algorithm to fine-tune a molecular structure generation model. The process is decomposed into two stages: pre-training, where graph neural networks and multilayer perceptrons are employed to build a molecular generation model, and fine-tuning, where goal-directed scoring functions are introduced. The fine-tuning phase uses a value network and reward-shaping mechanism to update the policy and optimize molecular generation. Their model surpasses Reinvent in generating valid molecule structures with +3% improvement.

Augmented hill-climb

Based on the combination of REINVENT and Hill-Climb, Thomas et al. [55] developed an RL-based method called Augmented Hill-Climb, which enhances sample efficiency. Their goal is to improve the efficiency of the SMILES-presented drug molecules generated. They found that the optimization ability is increased by 1.5 times and sample efficiency improved by 45 times compared to REINVENT, while still delivering interesting chemical outputs. On the other hand, it is more vulnerable to collapse, but it can be avoided by using diversity filters.

RM-GPT

Fan et al. [56] developed RM-GPT, a Recurrent Molecular-Generative Pretrained Transformer model designed to improve conditional molecular generation. RM-GPT enhances traditional GPT architectures by integrating LocalRNN and Residual Attention Layer Transformer modules. Therefore, RM-GPT captures local structural information and builds connections between effective attention blocks. This hybrid approach leverages the parallel computing capabilities of multi-head attention and generates drug-like molecules with certain properties and scaffolds. This leads to key advancements in the medical data field. First, LocalRNN enhances the model’s ability to capture localized structural medical insights, such as functional groups, improving its understanding of medical data. In addition, RealFormer strengthens the model’s effectiveness in generating stable, novel, lead-like compounds. Moreover, RM-GPT reliably produces lead-like molecules with desired properties or scaffolds, meeting essential requirements for practical drug development. Together, these innovations significantly advance the field of drug discovery by combining targeted insight acquisition with effective molecular generation. Fig. 2 represents a simple block diagram explaining the RM-GPT model architecture.

Fig. 2.

RM-GPT Architecture. RM-GPT employs a two-stage molecular generation process. In the first stage, users specify desired properties or scaffolds for the generated molecules. In the second stage, the trained model initializes generation with a single "C" token and iteratively samples from a probability distribution to produce subsequent tokens. This approach allows RM-GPT to generate molecules that closely match user-defined properties or exhibit high similarity to specified scaffolds, effectively combining conditional and unconditional generation techniques

Llamol

Dobberstein et al. [57] developed a new model called Llamol based on a transformer trained using Stochastic Context Learning (SCL) as a new training method. It is used to train the model for conditionally generated molecules. They wanted to ensure that the model perceives numerical value as well as the associated label. Their model handles single and multi-conditional organic molecular generation limited to four conditions, including three numerical properties and one token sequence, leading to valid molecular structures generation in SMILES notation. Llamol, due to its flexibility, incorporates various conditions into the generative process, making it adaptable for different molecular design tasks. Under all tested scenarios, the generated molecules showed satisfactory results with electro-active compounds, although the model excels in exploring organic chemical space. Furthermore, Llamol can easily incorporate new properties, enhancing its potential as a tool for de novo molecule design. Table 4 provides a comprehensive comparison of the works reviewed in the paper, highlighting the architecture and methodology used to optimize and enhance the molecular generation process

Table 4.

Overview of well-known generative models reviewed by the paper used for molecular generation enhanced with optimization techniques

Model	Validity & Novelty	Synthesizability	Data efficiency	Strengths	Limitations	Application contexts
REINVENT	High validity, good novelty	Moderate; needs post-filtering	Good with transfer learning	Strong in property optimization	May generate non-synthesizable molecules	Lead optimization, property design
JANUS	High novelty, diverse outputs	Not always synthesizable	Efficient with small data	Multi-objective optimization	Diversity and synthesis trade-off	Multi-property generation
MolVAE	Good validity, moderate novelty	Moderate; decoder-dependent	High (latent space)	Biology-based generation	Struggles with complex scaffolds	Disease-context design
MOLER	High validity, good novelty	Improved by graph constraints	Moderate	Preserves chemical rules	Complex training	Scaffold hopping, structure design
T&S Polish	Good validity, moderate novelty	Moderate	Moderate	Focus on topology	Diversity limitation	Scaffold optimization
EarlGAN	High novelty, moderate validity	Poor; needs filtering	Efficient with large data	Chemical space exploration	Synthesizability issues	De novo design, diversity sampling
MolGAN	High novelty, moderate validity	Poor; needs filtering	Efficient with large data	Chemical space exploration	Synthesizability issues	De novo design, diversity sampling
GenSMILES	Good validity, moderate novelty	Moderate	Efficient with SMILES	String-based, fast	limited 3D features	Rapid virtual screening
DIFFUMOL	High validity, high novelty	Good; improved with fragment-based design	Efficient with fragments	Excels in 3D structure-based design	Limited to structural fragments	Structure-based drug design
RM-GPT	High validity, good novelty	Good	High	Strong in scaffold-based generation	Prompt sensitivity	Property/scaffold-specific design
Llamol	High novelty, moderate validity	Unknown	Unknown	LLM-based generation; rapid exploration	Limited chemical constraints	Early-stage screening
DeepGraph MolGen	High validity, high novelty	Good	Efficient with graphs	Multi-objective RL; strong for property optimization and selectivity	Reward tuning is critical	Multi-property, selectivity design
DiffSMol	High validity, high novelty	Good; 3D structure -based	Efficient with 3D data	Excels in 3D molecular generation and property optimization	Limited binding poses novelty; biased toward known ligands	3D molecular generation
GCDM	High validity, high novelty	Good; 3D structure -based	Efficient with geometry	Geometry-complete diffusion; strong for complex molecules	Computationally expensive; slow for large-scale generation	3D molecular generation
GEOLDM	High validity, high novelty	Good; Latent diffusion	Efficient with the latent space	Geometric latent diffusion: combines geometric accuracy with efficient sampling	protein- or pocket-unaware; limited to small 3D molecules	3D molecule, property optimization
DTMol	High validity; strong novelty	Good	Moderate	Transformer-diffusion hybrid, so good control and efficiency More efficient than GCDM	Computationally expensive for large molecules	3D molecular generation; geometry- and property-conditional design

The table highlights the comparison between generative models across many dimensions, including validity, novelty, synthesizability, and data efficiency. Moreover, models' strengths and limitations are highlighted in addition to their application contexts

Variational autoencoder-based models

D-MolVAE

Disentangled representation learning aims to separate independent factors of variation within data, allowing for improved control and interpretability in generative models [58]. In molecular generation, disentangled representations facilitate more precise manipulation of molecular properties such as solubility, toxicity, and bioactivity. By isolating these attributes within the latent space, generative models can more effectively produce molecules that meet specified criteria, thereby streamlining molecular design for applications like drug discovery and material science. Yuanqi Du et al. [59] introduced the D-MolVAE framework, a graph-generative VAE designed to learn a disentangled representation of molecular structures. This disentanglement helps to explore how specific factors that encode chemical structures influence biological properties. The framework implements various mechanisms for disentanglement, leading to the development of several novel deep graph-generative models. The results demonstrated the efficacy of D-MolVAE in capturing the complex relationships between molecular structure and biological function.

Disentangled representation allows generative models to adjust molecular characteristics selectively. This is particularly advantageous in drug development, where certain properties like metabolic stability need to be optimized without compromising therapeutic activity. Researchers like to have demonstrated that disentangled VAEs can generate diverse molecular structures while ensuring that critical attributes remain within the desired range [60]. Disentangled representation learning contributes to the robustness of generative models by enabling generalization across varied chemical spaces. This is achieved by separating property-specific latent space, which allows the model to generate novel compounds without extensive retraining [61].

MOLER & graph polish

Existing molecular generation methods face challenges due to their reliance on iterative local expansion, often leading to molecules of arbitrary sizes and deviations in similarity and size compared to the target structure. Without a global fitness metric, generated molecules may lack alignment in molecular similarity and overall size, despite high scores for specific properties. Two main issues arise, including the trade-off between similarity and property optimization and the size consistency. Large size differences between generated and target molecules negatively impact similarity and property performance, as larger discrepancies reduce overall property optimization effectiveness.

Therefore, Tianfan Fu et al. [62] introduced Molecule-Level Reward functions (MOLER) to address the issue of not considering the whole molecule in the generation process but the substructure. Hence MOLER is used to force the input and the output to be the same size and similar to each other, as illustrated in Fig. 3a. Similarly, Chaojie Ji et al. introduced a molecular optimization approach known as "Graph Polish" [63]. This paradigm employs a heuristic optimization strategy guided by pairs of source and target molecules with desired properties. The process starts with predicting an optimization center within the input molecule, around which the surrounding regions are then optimized. They proposed a new learning framework, called Teacher and Student (T&S) Polish, to identify the dependencies involved in these optimization steps. The teacher component automatically captures and annotates the optimization centers and determines which molecular regions to preserve, remove, or modify. The student component then learns these patterns and applies them to optimize new molecules. Fig. 3b presents the teacher and student components. Table 2 summarizes the differences between MOLER and Graph Polish models using the ZINC and MOSES datasets. The comparison evaluates the percentage of molecules generated by each model that successfully exhibits the targeted properties, such as QED, DRD2, and LogP scores. Therefore, the generative models aim to generate molecules that should have a high QED score (good drug candidate), be active against DRD2 (high binding affinity to DRD2), and specific value for LogP. Notably, the T&S Polish model outperforms MOLER in the success rate of generated molecules.

Fig. 3.

a MOLER Architecture. The method employs a two-stage process: first, it trains a generative model on a dataset of molecules, and then it fine-tunes this model using reinforcement learning with a molecule-level reward function, enabling more effective exploration of chemical space and generation of molecules with desired properties. b Graph Polish Architecture. The overall framework of the proposed T&S polish consists of two components: the T component, which automatically identifies the optimization center and determines which parts of a molecule to preserve, remove, or add, and the S component, which learns these actions and applies them to generate new molecular structures

Table 2.

Comparative analysis between MOLER and Polish graph models using similar datasets

Model	Success rate of molecular generation (%)
	QED	DRD2	LogP
JTVAE + MOLER	43.2	40.01	45.24
VJTNN + MOLER	56.32	47.39	57.01
CORE + MOLER	57.32	49.47	57.93
T&S Polish	69.38	54.54	64.44

GenSMILES

Schoenmaker et al. [66, 59] tried to transform GenAI outputted invalid SMILES into meaningful ones. The study evaluated a SMILES corrector on four de novo molecular generation methods: RNN, target-directed RNN, GAN, and VAE, finding that the percentage of invalid outputs varied between 4% and 89% depending on the model. Post hoc correction using the SMILES corrector improved the validity of generated molecules, with models trained on multiple errors per input showing the best performance, correcting 60–95% of invalid outputs. The researchers implemented transformer models in PyTorch, adapting Ben Trevett’s Seq2Seq model. Input and output sequences were tokenized using TorchText, with the output sequence reversed. The SMILES tokenizer followed Olivecrona et al.’s approach [66], where most tokens were single characters, except for specific two-letter atom symbols and other complex tokens, creating a vocabulary of 101-110 tokens. The model’s architecture was based on Vaswani et al.’s transformer [19] but incorporated learned positional encodings and used the Adam optimizer without label smoothing, with a learning rate of 0.0005.

In another approach, Bhadwal et al. [66] introduced GenSMILES, a representation method designed to enhance the validity of SMILES strings in molecular generation tasks. GenSMILES addresses both syntactical and semantic issues inherent in traditional SMILES notation by employing derivative rules and simplified notations for branches and rings. This approach corrects common syntactical errors and mismatches in atom bonding, resulting in more accurate and valid molecular representations. Compared to SMILES and DeepSMILES, GenSMILES significantly improves the validity of generated molecules, achieving over 90% validity and a diversity score of 15while also expanding the exploration of chemical space. GenSMILES can be seamlessly integrated into existing generative algorithms and deep learning models, such as Recurrent Neural Networks and VAEs, without modifying the model environment.

In summary, Schoenmaker et al. [66] emphasized on correcting errors post-generation, whereas Bhadwal et al. [66] on preventing errors during the molecular generation process. Both approaches were found to significantly improve the validity of generated molecules which [66] were able to correct from 60 to 95% of invalid molecules using a post hoc SMILES correction method, while [66] achieved over 90% validity during the generation phase itself, without requiring any post-generation corrective steps.

Generative adversarial learning-based models

Huidong Tang et al. [67] developed EarlGAN, an advanced actor-critic RL agent-driven GAN designed for de novo drug design. Unlike traditional GANs used primarily for image processing, EarlGAN is tailored to generate SMILES strings by addressing the challenges of discrete molecular data. EarlGAN’s generator functions as an actor to produce SMILES strings, while the discriminator serves as a critic to evaluate them. The model employs autoregressive predictions at the atomic level, integrating instant rewards to lower the computation time, global-level rewards to provide molecular consistency, and information entropy maximization to enhance diversity in generated samples.

Jiayi Fan et al. [68] explored the influence of various factors on the validity score of molecular generation using a Molecular Generative Adversarial Network (MolGAN) architecture. Initially, the authors demonstrated that a basic Generative Adversarial Network (GAN) can generate valid molecular structures and that incorporating a reward network with the GAN in a reinforcement learning framework further enhances the validity score. Then, the effectiveness of the approaches was evaluated, focusing solely on optimizing the validity score and strategies for maintaining this score while optimizing other chemical properties. Their findings indicated that optimizing the validity score, independently or alongside other chemical properties, requires careful consideration and tuning of multiple factors, including loss functions, hyperparameters, and training protocols. Fig. 4 and Table 3 compare the architecture and the performance of both models, respectively.

Fig. 4.

GAN-based model’s architectures with RL technique for molecular generation guidance. a EarlGAN Architecture. EarlGAN employs an actor-critic reinforcement learning approach with a generator (actor) and discriminator (critic), using moment rewards and entropy maximization to guide the autoregressive generation of realistic molecular structures at the atomic level. b MolGAN Architecture. A validity improvement method for MolGAN-based molecular generation by incorporating a graph convolutional network (GCN) as a discriminator to distinguish between valid and invalid molecular structures, enhancing the model’s ability to generate chemically valid molecules

Table 3.

Comparative analysis between the performance of MolGAN and EarlGAN

Model	Performance measure
	Validity	Novelty	Uniqueness	Diversity
MolGAN	98.1%	10.4%	94.2%	–
EarlGAN	94.07%	86.24%	70.04%	0.92

Diffusion-based models

Diffusion models have become a revolutionary class of generative models in computational chemistry, particularly for de novo molecular generation. Initially, they were developed for image generation; however, they have been successfully re-purposed for the complex geometries and constraints of molecular design [71, 72]. Their growing prevalence is due to their ability to accurately capture high-dimensional multimodal distributions, which is crucial for molecular generation, while achieving validity, novelty, uniqueness, having correct stereochemistry, and being conformationally stable.

At a fundamental level, diffusion models are trained to invert a stochastic noise process over input data, consistently recovering valid molecular structures from noisy random input [22]. This step-by-step sampling process is different from the direct decoding of GANs or VAEs, resulting in better mode coverage and sample diversity. In the context of molecular design, this mechanism permits diffusion models to mimic naturally the delicate interplay between structural flexibility and physical realism. In contrast to SMILES- or graph-based generators that ignore spatial information, diffusion models are suitable to encode and reproduce 3D molecular conformations, making them highly valuable for structure-based drug design.

Models like PMDM and DiffSMol use protein pocket details and ligand shapes, creating molecules that bind well and have valid physical and chemical traits. They perform better than older methods in terms of how fast and how good the molecules are. Other models, such as Geometry-Complete Diffusion Models (GCDM) and EQGAT-diff, generate better 3D molecules that are real, stable, and large by adding geometric and chemical details using datasets like QM9, GEOM-Drugs, and Crossdocked datasets [73, 74]. In another study, Xu et al. [75] designed GEOLDM (Geometric Latent Diffusion Model) that leverages a latent-space formulation with geometry-aware constraints to enhance sampling speed and quality. In addition to generating 3D structures, they maintain the chemical validity of the generated molecules, outperforming traditional generative techniques in validity, uniqueness, and geometric fidelity. In terms of optimization, diffusion models can be directed toward certain chemical properties. This happens through conditional generation, sampling guided by gradients, or reward shaping that takes ideas from reinforcement learning. For example, conditional diffusion frameworks allow pharmacophore rules or property scores (e.g., logP, QED, DRD2) to be included directly in the generation process. Additionally, new methods in score-based guidance use learned energy functions or predictive models to move sampling toward desired chemical areas [76]. Chen et al. [77] recently combined diffusion with VAEs to create two-stage models. In these models, VAEs first encode molecular features into a basic hidden space. A diffusion decoder then improves this space, making samples more realistic and speeding up how quickly they come together. In terms of benchmarking, diffusion-based molecular models are evaluated using standardized benchmarks that capture structural accuracy, diversity, and biological relevance. PubChem3D, GEOM-Drugs, and QM9 assess 3D conformation accuracy, while MOSES and GuacaMol measure validity, novelty, uniqueness, and FCD. Binding affinity and RMSD-based docking scores evaluate performance in structure-based tasks, with PDBbind and Pocket2Mol used for ligand–receptor interaction modeling.

Hybrid models

Nigam et al. [78] presented JANUS, a genetic algorithm enhanced by parallel tempering that propagates two populations for exploration and exploitation to optimize molecular design while decreasing property evaluations. JANUS incorporates a deep neural network for molecular property approximation, utilizing active learning for improved sampling. It employs the SELFIES representation and the STONED algorithm for efficient structure generation. JANUS surpasses other models in inverse molecular design tasks, achieving better results. However, many generated molecules are synthetically infeasible, highlighting the need to consider synthesizability when evaluating generative models. Figure 5 shows the architecture of the JANUS model.

Fig. 5.

JANUS Architecture. Schematic representation of JANUS, featuring parallel populations with distinct genetic operators: an exploitative population driven by molecular similarity and an explorative population guided by a deep neural network for property estimation

Peng and Zhu [79] developed DIFFUMOL, a diffusion-based molecular generation model designed to fill the gaps found in existing deep generative models, such as low novelty, validity, and difficulty in generating molecules with desired properties and scaffolds. During the generation process, DIFFUMOL selectively adds Gaussian noise to preserve conditional features, guiding the creation of molecules with desired properties as illustrated in Fig. 6. DIFFUMOL enables finer tuning over molecular generation compared to one-step methods like GANs or autoencoders by utilizing the Markov chain concept to learn to reverse the diffusion process and optimize a variational lower bound.

Fig. 6.

DIFFUMOL Architecture. DIFFUMOL uses Gaussian noise to be added to the partial space iteratively to guide the molecular generation process

Using another strategy, Bhowmik et al. [80] tried to mitigate the limitations of traditional heuristic-based inverse design methods by using GenAI models in drug discovery, polymer science, and material science. However, GANs have challenges such as mode collapse, which hinders their effectiveness by limiting structural diversity. Therefore, they introduced a hybrid architecture that integrates masked language models with GANs. This approach leverages the strengths of both models to improve molecular generation, demonstrating improved performance compared to standalone language models, particularly in optimizing properties and generating novel molecules.

Based on conditional generation, Kim et al. [70, 74] presented an RL-guided combinatorial chemistry approach shown in Fig. 7, a rule-based molecular design method driven by trained policies for identifying molecular fragments to construct target molecules. The model can generate all possible molecular structures from fragment combinations, leading to the discovery of unknown molecules with superior properties. In experiments targeting molecules with extreme properties, the model identified 1315 molecules that hit the target and 7629 five-target-hitting molecules out of 100,000 trials, outperforming probability distribution-based models, while ensuring 100% chemical validity for all generated molecules.

Fig. 7.

RL-guided combinatorial chemistry Architecture

Table 4 shows a comparison between the reviewed generative models across many dimensions, including validity, novelty, synthesizability, and data efficiency. Additionally, the table highlights the strengths and limitations and application of each model. Table 5 provides a comprehensive comparison of the works reviewed in the paper, highlighting the architecture and methodology used to optimize and enhance the molecular generation process. Additionally, Fig. 8 summarizes all the utilized approaches and their base GenAI models.

Table 5.

Overview of recent studies using generative models for enhancement generation process techniques

References	GenAI model	Input data	Aim	Performance measure
Morgan Thomas et al. (2022) [55]	RNN & Transformer	Drug-like small organic molecules (GuacaMol dataset)	Enhance the sample-efficiency	Docking score & uniqueness
Nigam et al. (2022) [78]	JANUS (GA + DNN)	ChEMBL dataset	Inverse molecular design using genetic algorithm	Success, novelty, and diversity
Yuanqi Du et al. (2022) [59]	VAE	QM9, ZINC, MOSES and ChEMBL datasets	Generating small molecules based on disentanglement learning	Validity and uniqueness
Huidong Tang et al. (2023) [67]	GAN	QM9 and ZINC datasets	Design enhanced reinforcement learning	Validity
Jiayi Fan et al. (2023) [68]	GAN	QM9 dataset	Improve the validity score of the generated molecule	Validity
Tianfan Fu et al. (2022) [62]	MOLER (Encoder/Decoder)	ZINC database	Propose a novel molecule optimization paradigm (MOLER)	Success rate
Chaojie Ji et al. (2023) [63]	T&S Polish	ZINC database	Propose a novel molecule optimization paradigm (Teacher & Student Polish)	Success rate
Schoenmaker et al. (2023) [66]	RNN target- directed, VAE, and GAN	ChEMBL dataset	SMILES generation Enhancement	Uniqueness, novelty, similarity, and KL divergence
Bhadwal et al. (2023) [69]	RNN + VAE	ZINC, PubChem, and Tox21 datasets	SMILES generation Enhancement	Validity, uniqueness, and FCD
Loeffler et al. (2024) [53]	Transformer & RNN	Drugs (SMILES)	Design educational platform (Reinvent 4) for small molecular generation	Diversity
Sha and Zhu (2024) [54]	Gated Graph Neural Network (GGNN)	ChEMBL, ZINC, and GDB-13 dataset	Goal-directed molecular generation	QED
Bhowmik et al. (2024) [80]	GAN	QM9 dataset	Enhancing molecular design efficiency	QED, number of atoms, solubility, generated molecules, number of novel, number of accepted molecules, and the fitness function
Peng and Zhu (2024) [83]	Diffusion & Transformer	MOSES and GuacaMol dataset (ZINC)	Fine-grained molecular generation using attended diffusion model	Validity, uniqueness, and novelty
Fan et al. (2024) [56]	RNN + Transformer	ZINC database	Enhance the generative ability of molecular GPT model	Validity, uniqueness, and MAD
Kim et al. (2024) [70]	RL-CC	ChEMBL database	Molecule design	Validity and uniqueness
Dobberstein et al. (2024) [57]	Transformer	Drug-like molecules (OrganiX13)	Design a generative model based on Stochastic Context Learning	Interval novelty, uniqueness, validity, and mean average deviation
Tuan Le et al. (2023) [73]	Diffusion	QM9 and GEOM-Drugs	3D molecule structure generation	Atomic and molecular stability, diversity/similarity measures, validity, and Uniqueness
Morehead et al. (2023) [74]	Diffusion	QM9 and GEOM-Drugs	3D molecule structure generation	Diversity, QED, SA, molecule energy, and Lipinski
Xu et al. (2023) [75]	Diffusion	QM9	3D molecule structure generation	Validity and Diversity

The table highlights the types of generative models used, the input datasets, performance measures, and the aim of the studies

Fig. 8.

Summarized GenAI techniques used for the optimization of the generation process

Case studies

Generative molecule design has offered the theoretical basis and tools for exploring chemical space and optimizing molecular structures. A number of new structures have been reported [81–84]. However, few of these have found the way to successful real-world application and made tangible contributions to lead molecule discovery and optimization. The field has recently started to move beyond computational novelty and extended to molecule synthesis and discovery. One of the earliest AI-designed drug in clinical trials is the work reported by Zhavoronkov et al. were they developed deep generative model, generative tensorial reinforcement learning (GENTRL), for de novo small-molecule design to discover potential inhibitors of discoidin domain receptor 1 (DDR1) [85]. In their work, the generative model accounted for medicinal chemistry constrains to guarantee novelty, feasibility of synthesis, and biological activity. They have also used in-silico predictions, such as docking simulations, routine medicinal chemistry filters, to iteratively refine candidates. In addition, the team had an in-house lab facility for chemical synthesis and biological testing to validate and optimize molecular generation. However, long-term clinical validation remains pending.

Another successful case study reported by Gómez-Bombarelli et al. [86] who developed a method to convert discrete representations of molecules between a multidimensional continuous representation that allows the generation of new molecules for efficient exploration. Their model has optimized latent space, and improved molecule properties. More recently, Haotian Teng et al. have introduced a novel diffusion model, DTMol, to represent the first application of diffusion transformer architecture in solving pocket-based molecular docking [87]. Their model outperforms existing methods and experimentally confirmed JAK2 inhibitors through TR‑FRET assays with a 77.6% top‑1 docking success rate [87]. Despite remarkable success, the study synthesis and broader ADMET profiling was not performed. Furthermore, graph-based and diffusion models have advanced conditional molecular generation and pose prediction. For example, Gabriele C et al. have developed DiffDock that applies diffusion modeling directly to ligand pose generation and obtained a 38% top‑1 success on PDBBind offering calibrated confidence measures [88]. Meanwhile, Yufei H. et al., introduced Re-Dock as a novel diffusion bridge generative model extended to geometric manifolds and proposed energy-to-geometry mapping inspired by the Newton–Euler equation [89].

In real-world drug design scenarios, a conditional equivariant diffusion model (PMDM) was used for lead optimization against the SARS-CoV-2 main protease and Cyclin-dependent Kinase 2 (CDK2) [88]. Notably, molecules generated by PMDM were synthesized and tested in vitro, with some showing improved activity against CDK2. Graph-based generative models like GraphDF [90] have suggested viable COVID-19 target molecules, while REINVENT has demonstrated RL-based generation of optimized analogs for existing drugs like osimertinib [66]. Junction Tree VAE (JT-VAE) generated synthetically accessible, target-active molecules for GSK3β and JNK3 [91], and DeepLigBuilder enabled structure-based 3D ligand generation in fragment-based design [92]. An unsupervised molecule-to-molecule generative framework successfully produced FDA-approved drugs such as Perazine and Clozapine [86], which is now deployed in personalized medicine laboratories for targeted drug delivery. Similarly, the SAMOA algorithm, a scaffold-constrained generative model based on recurrent neural networks, has been used in industrial lead optimization to design novel, active compounds for targets such as Dopamine Receptor D2 and MMP-1[85].

Challenges and limitations

Despite several efforts to improve the GenAI models used for molecule design optimization, challenges and limitations persist in this field, requiring rigorous examination and improvements. One major challenge is the huge complexity and vastness of chemical space, which hinders comprehensive exploration using AI models. There is uncertainty and randomness in predicting molecular properties and behaviors from structural data, and it can often result in unreliable and inaccurate outcomes for practical research, reducing the possibility of discovering real innovative compounds. This uncertainty and stochasticity are owed to the highly complex relationships between molecular structures and behavior, and the lack of solid understanding of it, potentially yielding different and unreliable results every time a prediction is made. Luckily, recent studies have proposed strategies to address this challenge. Integrating goal directed optimization and fragment-based methods into model training using uncertainty quantification methods and reinforced learning can guide the exploration of chemical space through focusing on subspaces aligned with synthetic and biological constraints. Additionally, the instability in training such as mode collapse and vanishing gradients leads to unreliable and low diversity outputs. Therefore, improved training techniques such as Wasserstein loss functions can stabilize the learning [93, 94].

Another major challenge faced in the field is the limited availability of quality and diverse data, restricting the training and performance of the model, especially for uncommon or recently discovered types of molecules. Several available datasets focus on common and well-reported molecules, disregarding new, rare, and less-reported molecules. This imbalance restricts the performance of the AI predictive models beyond common chemical domains such as orphan drug discovery and new material synthesis. In addition, some datasets contain errors, biases, insufficient information, inaccuracies, inconsistencies, and variations, which can affect the model training and result in unreliable predictions. Hence, strategies like data augmentation transfer learning and active learning are increasingly used to improve robustness and applicability. Data augmentation techniques can expand the existing datasets synthetically and improve model robustness and generalization [49]. For example, in molecular design, it can generate multiple realistic but novel representations of certain molecules to further enhance the diversity of training datasets, reduce overfitting, and improve prediction [95–97]. Transfer learning can also be used to allow the models to adapt to new domains using limited data [98, 99]. Furthermore, active learning is very important when experimental resources are limited as it prioritizes which molecules to synthesize or simulate next based on model uncertainty or expected information gain. It can help maximize learning from limited dataset and improve model accuracy efficiently over time [100].

Understanding why a particular molecular structure is predicted by an AI model is crucial for scientific validation and reliability. However, several reported AI models lack transparency and operate with an unclear decision-making process. There are other challenges faced when it comes to the transition from theoretical designs to real-world applications and synthesizing generated molecules. Synthesizability is a paramount concern as many molecules might not be feasible to make due to constraints like availability or starting material, reaction conditions, and scalability.

However, closer collaboration between computational and experimental researchers, as well as active learning frameworks that incorporate assay data, can bridge the gap. Furthermore, in addition to being synthetically feasible, the generated molecules must comply with specific rules depending on their class or application to define if they exhibit desirable properties like absorption, distribution, metabolism, toxicity, and biocompatibility. For example, Lipinski’s Rule of Five helps define drug-likeness in the area of drug discovery. However, these evaluation metrics (QED, SA score, etc.) do not prioritize chemical feasibility over biological relevance. For example, the generated molecule can have a high QED but no binding affinity leading to false positive candidates. Therefore, the solution is to use multi objective optimization techniques (Sect. "Multi-Objective Optimization Techniques"), incorporating binding affinity predictions which align model outputs with experimental goals. Moreover, several efforts should be dedicated to improving interdisciplinary collaboration, standardization of datasets, and advancements in validation methodologies. The development of more efficient algorithms for chemical space exploration, improved methods for synthesizability prediction, and enhanced model interpretability. Furthermore, other issues should be addressed including refining the approaches for handling data scarcity and establishing robust frameworks for empirical validation of AI-driven molecular design optimization.

Conclusion

The application of GenAI models into molecular design has enabled the exploration of vast chemical spaces which significantly improved the field of de novo compound generation, offering unprecedented efficiency for optimizing molecular structures. The current review highlights diverse approaches and techniques in applying AI for generating valid and functionally relevant molecule structures, like variational autoencoders, generative adversarial networks, and transformer-based architectures. These models have outperformed traditional methods in terms of efficiency and creativity. Looking forward, there are exciting possibilities for advancements in this field, especially in integrating domain-specific knowledge more deeply into the AI models. Developing such hybrid approaches offers the strengths of different architectures and refined methods for validating and experimentally testing generated molecules. Moreover, as these models grow more sophisticated, the related ethical considerations become more complicated making it more challenging to ensure the responsible use of AI in molecular design. Despite the exponentially increasing challenges, the rapid progress of innovation in this area suggests that it will continue to play a pivotal role in accelerating scientific discovery and technological breakthroughs.

Author contributions

T.K. and S.A.A. wrote the main manuscript text T.K. and S.A.A. prepared the figures. A.A., C.P., A.M.P., S.T.N., V.C. and V.K.T. reviewed the manuscript.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Competing interests

The authors declare no competing interests.