A Perspective on Foundation Models in Chemistry

Abstract

Foundation models are an emerging paradigm in artificial intelligence (AI), with successful examples like ChatGPT transforming daily workflows. Generally, foundation models are large-scale, pretrained models capable of adapting to various downstream tasks by leveraging extensive data and model scaling. Their success has inspired researchers to develop foundation models for a wide range of chemical challenges, from materials discovery to understanding structure–property relationships, areas where conventional machine learning (ML) models often face limitations. In addition, foundation models hold promise for addressing persistent ML challenges in chemistry, such as data scarcity and poor generalization. In this perspective, we review recent progress in the development of foundation models in chemistry across applications of varying scope. We also discuss emerging trends and provide an outlook on promising approaches for advancing foundation models in chemistry.

1. Introduction

Artificial intelligence (AI) has achieved remarkable success in fields such as computer vision (CV) and natural language processing (NLP), motivating scientists and researchers to explore its potential in chemical and materials science.¹ For instance, machine learning (ML) models are now used to predict molecular and material properties.²⁻⁶ Computational chemists use ML models as surrogate potentials, known as machine learning interatomic potentials (MLIPs), to accelerate computationally demanding ab initio molecular dynamics (AIMD) simulations.⁷⁻¹¹ Another growing area is inverse design, where generative models are used to directly generate novel molecules and compounds with desired properties.¹²⁻¹⁸ Beyond these, ML has been applied to predict drug interaction,^19,20 synthesis routes,²¹⁻²³ reaction products,^24,25 and analyze X-ray diffraction patterns.^26,27

One of the central challenges for ML in chemistry is the scarcity of large, labeled data.²⁸ Unlike CV and NLP, where vast amounts of annotated data are readily available, chemical and materials science datasets are often limited and require labor-intensive experiments or enormous computing resources. Moreover, applications like drug discovery or materials design require extrapolation to out-of-domain compounds, another limitation of deep learning models. These challenges limit conventional ML approaches at a practical level. In addition, traditional MLIPs often rely on deliberately curated datasets crafted with domain expertise to avoid extrapolation, limiting their transferability across different systems and thus restricting broader adoption.

Recently, the concept of foundation models has emerged as a new AI paradigm. Foundation models are large-scale, pretrained models that can adapt to a broad range of downstream tasks.²⁹ By training on vast datasets, foundation models learn general representations that can be shared across different tasks and domains. These models are then adapted to downstream tasks through transfer learning or finetuning. For instance, foundation models in NLP may handle text translation and summarization, while in CV, they are adapted for tasks like image classification and captioning. In the context of chemistry and materials science, a foundation model can be adapted to predict various properties of molecules and crystals,³⁰ or generate novel compounds with a desired property (Figure 1).¹⁶ Another application is a foundational MLIP, which can be transferred to a wide range of systems and can be further finetuned to a target system when necessary.³¹ Recently, large language models (LLMs) such as ChatGPT³² have been leveraged to solve chemical problems or finetuned for specific tasks.^33,34 Owing to their flexibility and generalizability, foundation models offer not only improved performance over models trained from scratch but also potential solutions to the aforementioned challenges such as data scarcity and robust extrapolation.³⁵⁻³⁸ Moreover, multimodal foundation models³⁹ in chemistry leveraging molecular structures and text would provide a user-friendly platform for text-based inverse design of molecules, which was beyond the capabilities of the conventional ML models.^40,41 However, research related to foundation models is evolving rapidly and extensively, making it challenging to track recent trends and identify promising approaches. Therefore, it is an appropriate time to summarize current efforts and provide an outlook on foundational models in chemistry.

Figure 1.

Overview of a foundation model in chemistry for property prediction, machine learning interatomic potentials, and inverse design.

The precise definition of the foundation model in chemistry has not been established, and it may differ depending on the way the scope of downstream tasks is considered. Broadly speaking, a pretrained prediction model that can be adapted to various property prediction tasks (e.g., band gap, formation energy, etc.) can be considered a foundation model for a property prediction domain. In a narrower sense, a foundation model should be able to span multiple applications such as prediction of properties and PES, generation of data, etc. While the latter may be considered closer to an ultimate form of the foundation model, studies on the former focusing on pretraining methods that work well in each domain are also crucial for developing pretraining strategies for the latter. In this perspective, therefore, we consider both definitions, referring to the former as a small foundation model and the latter as a big foundation model. First, we review the current progress in the small foundation model for three respective domains: (1) property prediction, (2) MLIP, and (3) inverse design (Table 1). Then, we look into big foundation models adaptable to multiple domains in the (4) multi-domain sections (Table 1). Before this, we provide an overview of common methodologies investigated for foundation models in chemistry. Finally, we outline future directions and opportunities.

Table 1. Summary of Foundation Models in Chemistry.

			Pretraining
Domain	Model	Architecture	Data	Methoda	Downstream task
Property prediction	GraphCL105	GIN106	ZINC15107 (2M)	CL (aug.)	Molecular property prediction
	MolCLR108	GCN,109 GIN106	PubChem110 (10M)	CL (aug.)	Molecular property prediction
	GraphMVP111	GIN,106 SchNet112	GEOM113 (50K)	CL (2D ↔ 3D)	Molecular property prediction
	Hu et al.114	GIN,106 GCN,109 GraphSAGE115	ZINC15107 (2M), ChEMBL116,117 (456K)	PL (node context), GL (node, edge), SL (property)	Molecular property prediction
	GROVER118	GTransformer	ZINC15,107 ChEMBL116 (total 11M)	PL (motif), GL (node, edge)	Molecular property prediction
	SMILES-BERT119	BERT63	ZINC120 (18M)	GL (SMILES)	Molecular property prediction
	ChemBERTa-2121	RoBERTa122	PubChem110 (77M)	GL (SMILES), SL (property)	Molecular property prediction
	MoLFormer123	Transformer62	PubChem110 (111M), ZINC124 (1B)	GL (SMILES)	Molecular property prediction
	MOFTransformer125	BERT63	In-house hMOF (1M)	PL (property)	MOF property prediction
	CT126	CGCNN96	Matminer127 (152K), hMOF128 (275K)	CL (aug.)	Materials property prediction
	CrysGNN35	CGCNN,96 CrysXPP,129 GATGNN,130 ALIGNN97	OQMD131 (661K), MP132 (139K)	CL (crystal system), PL (space group), GL (node, connectivity)	Materials property prediction
	DSSL133	DeeperGATGNN134	MP132 (138K)	CL (aug.), PL (microproperty), GL (node)	Materials property prediction
	LLM-Prop135	T5136	MP132 (144K)	GL (text75)	Materials property prediction
Machine learning interatomic potentials	M3GNet137	GNN	MPF.2021.2.8132 (187K)	SL (E, F, S)	MD simulations and structural relaxation
	CHGNet138	GNN	MPtrj132 (1.58M)	SL (E, F, S, M)	MD simulations and structural relaxation
	ALIGNN-FF139	ALIGNN97	JARVIS-DFT140 (307K)	SL (E, F, S)	E–V calculation, structural relaxation, structure search
	PFP141	TeaNet142	PFP molecular dataset (6M), PFP crystal dataset (3M)	SL (E, F, C)	MD simulations, molecular adsorption, order–disorder transition, material discovery for catalysts
	MACE-MP-031	MACE58	MPtrj132 (1.58M)	SL (E, F, S)	35 applications including water, catalyst, MOF, battery cell, etc.
	MACE-OFF23143	MACE58	OFF23 (1M)	SL (E, F)	Dihedral scans, MD simulations of molecular crystals, organic liquids, etc.
	GNoME potential144	NequIP57	GNoME (89M)	SL (E, F)	Crystal structure search
	SevenNet145	NequIP57	MPtrj132 (1.58M)	SL (E, F, S)	Melt-quench simulation
	MatterSim36	M3GNet,137 Graphormer65	In-house data (3M, 17M)	SL (E, F, S)	Calculation of thermodynamics, lattice dynamics, and mechanical properties
	eqV2146	EquiformerV2147	OMat24 (118M)	SL (E, F, S), Denoising	Structural relaxation
Inverse design	MatterGen16	Diffusion	Alex-MP-20132,148 (607K)	GL (A, X, L)	Generation of crystals with target property
	GP-MoLFormer149	MoLFormer123	PubChem150 (111M), ZINC124 (1B)	GL (SMILES)	Generation of molecules with target property
	CrystalLLM151	Pretrained LLaMA-2152	–	–	Generation of crystals with target property
Property prediction & MLIP	Zaidi et al.153	GNS154	PCQM4Mv2155 (3.4M)	Denoising	Energy, force and molecular property prediction
	GeoSSL-DDM156	PaiNN59	Molecule3D157 (1M)	Denoising	Force and molecular property prediction
	ET-OREO158	TorchMDNet66	MD17159 (3.5M), ANI1-x160 (5M), PCQM4Mv2155 (3M), poly24 (3.5M)	Denoising, SL(F)	MD simulations, molecular property prediction
	3D-EMGP158	EGNN54	GEOM-QM9113 (100K)	Denoising	Force and molecular property prediction
	Frad161	TorchMDNet66	PCQM4Mv2155 (3.4M)	Denoising	Energy, force and molecular property prediction
Property prediction and inversedesign	KV-PLM162	BERT63	S2orc163 (0.3M papers, 1B tokens), PubChem164	Multimodal learning (SMILES, text)	Molecular property prediction, reaction classification, SMILES-description retrieval
	MoMu165	SciBERT,166 KV-PLM,162 GraphCL105	PubChem,164 S2orc163 (15K graph-text pairs)	Multimodal learning (graph, text)	Graph-description retrieval, molecule captioning, text-to-graph generation, molecular property prediction
	MolFM167	KV-PLM,162 GraphMVP,111 TransE168	PubChem,164 S2orc,163 DrugBank169 (15K graph-text pairs), knowledge graphs (E49K, R3.2M)	Multimodal learning (graph, text, knowledge graph)	Graph-description retrieval, molecule captioning, text-to-graph generation, molecular property prediction
	MoleculeSTM40	MegaMolBART,170 GraphMVP,111 SciBERT	PubChemSTM164 (281K structure-text pairs)	Multimodal learning (SMILES/graph, text)	Structure-text retrieval, text-based molecule editing, molecular property prediction
	SPMM41	BERT63	PubChem171 (50M)	Multimodal learning (SMILES, property)	Property-to-SMILES generation, molecular property prediction, reaction prediction
	ChemDFM172	Pretrained LLaMa-13B173	Chemical books (1.4K), papers (3.9M), general text	Language modeling	Molecule recognition, text-to-SMILES generation, molecular property prediction, reaction prediction
	nach0174	T5136	Text from PubMed (13M, 355M tokens), USPTO (119K, 2.9B tokens), ZINC (100M, 4.7B tokens)	Language modeling	14 tasks including molecular property prediction, reaction prediction, text-to-SMILES generation, etc.
	Jablonka et al.175	Pretrained GPT-332	–	–	Molecular/materials property prediction, text-to-SMILES generation
	AtomGPT176	Pretrained GPT-2177	–	–	Materials property prediction
		Pretrained Mistral 7B178	–	–	Text-to-materials generation

^aCL: contrastive learning; aug.: augmentation; PL: predictive learning; GL: generative learning; SL: supervised learning; E: energy; F: forces; S: stress; M: magnetic moments; A: atom types; X: positions; L: lattice.

2. Methodology

2.1. Models

2.1.1. Graph Neural Network

Conventional machine learning models often require hand-crafted descriptors to capture input–output relationship.^5,42−46 Designing an effective descriptor requires domain expertise on the target task, and often involves a labor-intensive feature selection process to identify the best-working descriptors. In the case of MLIPs, most conventional approaches are descriptor-based, with the descriptors designed to capture information on the local environment constructed by the elements specific to the target system.^47,48 As a result, this approach limits the transferability of trained MLIPs to other systems.

In contrast, deep learning models using neural networks are known to learn the representation of the input data that best describes the underlying relationship between input and output, generally outperforming the descriptor-based models.^49,50 Among these, graph neural networks (GNN) have gained significant attention for their high expressivity and suitability for molecular data,⁵¹ making them promising candidates for foundational models in chemistry. In a GNN, molecules or crystals are represented as graphs G = (V, E), where atoms and bonds are treated as nodes V and edges E, respectively.⁵² The graphs are typically featurized with node features , such as atom types, and edge features , such as bond types and lengths, where i and j denote the node indices of a graph.⁵¹ Node features are then updated via a message-passing framework, where the message is constructed based on the environment of the central node or edge.⁵³ Specifically, the message for node i at the (t + 1)^th message-passing step is obtained in general as

1

where N(i) denotes the number of nodes in the graph, and M(t) is a message function. Then, node features are updated via an update function U_t:

2

After T message-passing steps, a readout function R is applied to the updated node features to obtain a graph-level feature g:

3

Then g is projected onto multilayer perceptrons (MLP) to output a prediction. It should be noted that the update functions M_t and U_t, and the readout function R can be approximated by learnable neural networks, enabling the effective learning of complex interactions between atoms. Moreover, the recent equivariant GNN enables the use of vectorial features richer in geometric information such as positions, thereby enhancing the expressive power of the GNN.⁵⁴⁻⁵⁶ As a result, they have been actively explored as MLIPs, where the predictions (e.g., energy and forces) are sensitive to the geometries of molecules and crystals.⁵⁷⁻⁶⁰

2.1.2. Language Model

A language model is a key area in NLP that aims to learn and predict the likelihood of word sequences.⁶¹ The development of BERT by Google, built on transformer architecture,⁶² marked the beginning of the foundation model era by showcasing that a single model could perform multiple tasks.^29,63 Furthermore, scaling up language models in both model size and data volume has revealed new capabilities, or “emergent abilities”, enabling them to solve complex tasks that were previously out of reach.⁶⁴ As a result, large language models (LLMs) like ChatGPT²⁹ have gained widespread popularity in various fields where tasks can be expressed in human languages, such as language translation, research assistance, and more.

An autoregressive language model such as GPT aims to predict the next token y in a sequence given a context X.⁶¹ The context consists of the preceding tokens x₁, x₂, ..., x_t–1, where t denotes the current position in the sequence. The model is trained by maximizing the conditional probability of the token sequence, expressed as P(y|X) = P(y|x₁, x₂, ..., x_t–1). Using the chain rule, this objective can be further decomposed into the product of probabilities:⁶¹

4

where T is the sequence length.

One of the most successful language models, a transformer, leveraged self-attention mechanism eliminating the need for recurrent neural networks (RNNs) or convolution.⁶² Specifically, the attention is computed by

5

where Q, K, and V are the query, key, and value matrices, respectively. To capture rich and intricate information from the input, multi-head attention is used in the transformer, which applies the attention mechanism across h independent subspaces:⁶²

6

7

where , , , and , d_k and d_v are dimensions of query/key and values, respectively, and d_model/h = d_k. In addition, since the transformer does not rely on RNNs or convolution, positional encoding (PE) is used to make aware of the position of a token within the sequence:⁶²

8

The transformer is known for its effectiveness in capturing the relative importance and long-range dependencies of each element in a sequence, making it the mainstream model in language models today. In fact, this has also led to the development of GNNs inspired by transformer architecture to effectively learn interactions between nodes.⁶⁵⁻⁶⁷

With the success of language models in NLP, efforts to apply these models in chemistry have grown extensively.⁶⁸⁻⁷¹ A crucial aspect of this application is identifying effective string-based representations for input. Molecules can be readily represented as a string such as the Simplified Molecular Input Line System (SMILES)⁷² and Self-referencing embedded strings (SELFIES),⁷³ enabling active use of language models in chemistry.⁷⁴ For example, transformers have been applied almost directly with minimal modification to predict chemical reactions represented by the rearrangement of SMILES from reactants to products.²⁵ Conversely, in materials science, the lack of a standardized text-based representation for crystalline structures limits the broader application of language models. However, tools such as Robocrystallographer⁷⁵ and ChemNLP⁷⁶ have been developed to generate text descriptions for crystals. Additionally, Xiao et al. proposed a Simplified Line-Input Crystal-Encoding System (SLICES), which offers a more compact and invertible string representation for crystals.^77,78 These advancements are driving increased research into language models for crystalline materials. As a result, large language models have become another key foundation model in the field of chemistry.

2.2. Pretraining Methods

2.2.1. Self-Supervised Learning

A prominent pretraining approach for the foundation model is self-supervised learning (SSL), which leverages the inherent structure of data as a source of supervision without requiring human-labeled data.⁷⁹ While supervised learning relies on labeled data which is often costly and time-consuming to generate, SSL enables the use of abundant unlabeled data to learn intrinsic data representations that can be finetuned for downstream tasks with a smaller amount of labeled data. Indeed, most successful foundation models including those based on transformer architectures were pretrained using SSL.

SSL is often categorized into three approaches: contrastive, predictive, and generative learning.^79,80 Contrastive learning has been actively explored in CV and NLP domains.,⁸¹⁻⁸⁴ where representations are paired as either positive or negative, with the model learning to maximize similarity between positive pairs while pushing negative pairs apart. The pairing method is crucial and must be designed carefully to ensure the learned representation is relevant to downstream tasks and avoids negative transfer.^80,81 Predictive learning is known as a method that predicts self-generated informative labels from the data, while generative learning refers to reconstructing a graph or text.^79,80 While contrastive learning focuses on inter-data information, these two learning schemes aim to extract intra-data features.⁸⁰ In the following sections, we review the approaches to adapting these SSL methods for foundation models in chemistry, and for more general and theoretical discussions and benchmarks on SSL, we refer readers to refs (30, 79, 80).

2.2.2. Multimodal Learning

Multimodal learning aims to capture complementary information from multiple sources or modalities, similar to how humans perceive voice while seeing the speaker.^85,86 This approach has achieved remarkable success in vision-and-language pretraining (VLP).⁸⁷⁻⁹⁰ For instance, Contrastive Language-Image Pretraining (CLIP) learns a common embedding space for text and images by jointly training respective encoders,⁸⁷ such as a transformer for text and a convolutional neural network for images.⁹¹ Specifically, contrastive learning is used to maximize the similarity of embeddings from the real text-image pairs while minimizing it for incorrect pairs, enabling zero-shot image classification by comparing the similarity between the query image and label texts. The success of VLP has encouraged researchers to explore multimodal learning in chemistry, becoming a promising approach to the development of big foundation models capable of handling diverse downstream tasks across two modalities (Section 3.5.1). More discussion on multimodal learning for foundation models in materials science can be found in Takeda et al.,⁹² to which we refer interested readers.

3. Applications

3.1. Property Prediction

Obtaining molecular and material properties through experimental or computational methods like quantum mechanical calculations is resource-intensive, motivating the development of machine learning models to predict various properties by learning structure–property relationships from data.⁹³⁻⁹⁵ However, deep learning requires large datasets, which are often limited in chemistry, especially for experimental data. In addition, machine learning models often struggle with extrapolating to unseen data, a frequent requirement in chemistry such as with newly designed materials. While advancements in architectures and techniques have improved predictive performance in chemistry,^60,96−98 challenges of data scarcity and generalization remain unresolved. Transfer learning partially addresses these issues by reusing representations from large datasets, yet effective transfer depends on close alignment between pretraining and target tasks, requiring domain expertise to avoid negative transfer.⁹⁹⁻¹⁰¹ By contrast, foundation models seek to learn universal representations adaptable across diverse properties, holding the potential to address data limitations and improve generalizability, even in low-data and out-of-distribution (OOD) domains. In this section, we review the progress in small foundation models for property prediction, focusing on various self-supervised pretraining strategies.

3.1.1. Contrastive Learning

One of the early studies on contrastive learning for graph data was Deep Graph Infomax (DGI),¹⁰² inspired by Deep InfoMax⁸³ from CV. DGI generated pairs by comparing local and global views of the graph. Specifically, it learns node embedding in such a way that the embedding of the node and its parent graph (positive pair) becomes similar while the node and the other graphs (negative pair) are dissimilar. InfoGraph¹⁰³ adopted a similar approach, with a more focus on graph-level embeddings. When combined with supervised learning, InfoGraph outperformed purely supervised learning on all 12 targets of the QM9¹⁰⁴ dataset and surpassed semi-supervised baselines on 11 targets.

While these methods focused on contrastive learning of local and global features within graphs,⁸⁰ more recent works have explored contrastive learning of graph-level embeddings via data augmentation (Figure 2).^{105,108,111,179−181} For instance, GraphCL¹⁰⁵ applied data augmentation to graph data for contrastive learning of graph-level embeddings. Specifically, GraphCL introduced four data augmentation techniques: node dropping, edge perturbation, attribute masking, and subgraph sampling. During training, one augmentation method was randomly applied, treating the augmented and original graphs as a positive pair, while pairing the augmented graph with other graphs as negatives. After transfer learning, GraphCL achieved state-of-the-art (SOTA) performance on 5 of 9 tasks, including MoleculeNet¹⁸² classification task. Given the importance of choosing suitable augmentation techniques for downstream performance, the same group further proposed an automated framework to optimize augmentation selection for pretraining.¹⁷⁹

Figure 2.

Contrastive learning proposed in MolCLR. Reprinted with permission from ref (108). Copyright 2022 Springer Nature. a. Schematic of contrastive learning and b. proposed augmentation methods for molecular graphs. c. Learned features via contrastive learning visualized by t-SNE.

More recently, Wang et al. introduced MolCLR,¹⁰⁸ a contrastive learning framework for molecular graphs trained on an enlarged dataset of 10 million molecular graphs from PubChem.¹¹⁰ Benchmarking on MoleculeNet classification and regression tasks showed that MolCLR achieved significant improvements compared to the supervised counterparts. Additionally, t-SNE visualization of the learned features showed that molecules with similar topologies and functional groups were clustered together (Figure 2c), demonstrating MolCLR’s effectiveness in capturing meaningful molecular representations.

Recognizing that 3-dimensional (3D) molecular geometry provides richer information than 2-dimensional (2D) molecular topology, Liu et al. proposed GraphMVP, a framework that utilizes both 3D geometry and contrastive learning to enhance representation learning.¹¹¹ In GraphMVP’s pretraining stage, each molecule’s 2D topology and 3D geometry were paired as positives, with negatives defined as pairs of different molecules. In the downstream task, only 2D molecular graphs were used to predict properties, with the test dataset differing from the pretraining dataset. When combined with conventional 2D SSL, GraphMVP outperformed prior SSL approaches across all 8 classification and 6 regression tasks. Moreover, the authors suggested that 3D geometry served as privileged information, improving separability among molecules and accelerating convergence as explained by VC theory.¹⁸³

Contrastive learning has also shown promise for crystalline materials. Crystals differ from molecules by combining a broader range of elements with diverse symmetries, which may necessitate different approaches. One of the earliest studies by Magar et al. proposed three augmentation methods based on random perturbations, atom masking, and edge masking.¹²⁶ Using CGCNN⁹⁶ as a baseline model, this approach showed improvements in 7 out of 9 Matbench⁹⁴ benchmarks. CrysGNN, proposed by Das et al., took a different approach by defining crystals of the same (different) crystal system as positive (negative) pairs.³⁵ The rationale behind this lies in the fact that some electronic and optical properties such as band gap and dielectric constant depend on the spacegroup and crystal structures.

3.1.2. Predictive and Generative Learning

In chemistry, the labels for predictive learning can be contextual information of a node,¹¹⁴ a motif within a molecule such as a functional group,¹¹⁸ chemical properties of atoms such as electronegativity, number of valence electrons, and covalent radius,³⁵ or the space group.¹³³ For generative learning, a typical approach is to mask nodes or/and edges and reconstruct them^114,133 or to reconstruct the entire molecule or crystal,¹⁸⁴ which is essentially generative modeling.

Predictive and generative learning are often applied in a multi-task fashion and are sometimes integrated with contrastive or supervised learning.^{35,114,118,133} For example, Hu et al. proposed a general framework to learn representations from both local and global aspects of the molecular graph.¹¹⁴ The local features were learned through predictive learning on the node context and generative learning by reconstructing masked nodes and edges, while the global features were learned via supervised learning of molecular properties. Rong et al. took a similar approach but differed at the graph-level representation learning, where they suggested predicting the graph-level motif such as functional groups, rather than supervised learning.¹¹⁸

For crystalline materials, CrysGNN, mentioned earlier in Section 3.1.1, combines generative and predictive learning with contrastive learning.³⁵ This model reconstructs node features and connectivity at the node level and uses space group prediction to learn graph-level embeddings. In downstream tasks on the Materials Project (MP)¹³² and JARVIS-DFT¹⁴⁰ datasets, CrysGNN improved the performance of all four models tested across various properties owing to the synergy of these SSL methods. CrysGNN also demonstrated improvements even on small experimental datasets, implying its practical utility. More recently, Fu et al. proposed using atomic-level microproperties as predictive labels in addition to contrastive learning as shown in Figure 3a.¹³³ Their approach is motivated by the fact that macro-properties such as elastic properties or band gaps depend on the ensemble of microproperties of local atoms, such as atomic stiffness or valence electrons. However, selecting relevant pretraining labels for downstream tasks requires domain knowledge and must be done carefully.

Figure 3.

a. Predictive and generative learning on crystal graphs. Reprinted with permission from ref (133). Copyright 2024 American Chemical Society. b. Overview of MoLFormer and c. its attention maps, with linear attention capturing the 3D distance information from SMILES. Reprinted with permission from ref (123). Copyright 2022 Springer Nature.

Language models also have been employed in terms of predictive and generative learning. For example, Wang et al. pretrained BERT through a Masked SMILES Recovery task, training the model to recover randomly masked SMILES.¹¹⁹ After finetuning, this model outperformed the SOTA on tested three datasets, demonstrating the effectiveness of their pretraining approach. Similarly, Zhang et al. used BERT with SMILES augmentation by varying starting atoms and traversal orders, which were then masked for pretraining.¹⁸⁵ The finetuned model achieved SOTA performance on most of the 60 molecular prediction tasks, including ADMETlab¹⁸⁶ and MoleculeNet.

Chithrananda et al. proposed ChemBERTa, and first systematically demonstrated the potential of transformers in molecular representation learning by exploring data size, tokenizer methods, and string representations.¹⁸⁷ The following work by Ahmad et al. developed ChemBERTa-2,¹²¹ investigating pretraining strategies such as masked language modeling and multi-task regression with labels computed using RDKit,¹⁸⁸ without the need for extra experiments. ChemBERTa-2 pretrained with multi-task regression outperformed the original model and SOTA models on some MoleculeNet tasks.

Ross et al. introduced MoLFormer, a transformer-based model pretrained on the PubChem¹¹⁰ and ZINC¹²⁴ datasets, encompassing 1.1 billion SMILES using masked language modeling (Figure 3b).¹²³ MoLFormer was benchmarked on various downstream tasks, including 2D and 3D molecular datasets like MoleculeNet and QM9, outperforming baseline and other language models and even some GNN models. Analysis of MoLFormer’s attention maps revealed that the model effectively captured molecular structural features from SMILES (Figure 3c). Kang et al. proposed MOFTransformer, a multimodal transformer for metal–organic frameworks (MOFs), combining local features derived from CGCNN with global energy grid features.¹²⁵ The pretraining task was predictive learning to predict MOF properties such as topology and void fraction and to classify metal cluster-organic linkers. MOFTransformer achieved SOTA across diverse properties, including gas adsorption, diffusion, and electronic properties, surpassing baselines including CGCNN.

Rubungo et al. developed LLM-Prop, a language model pretrained on text descriptions of crystal structures, then finetuned to predict crystal properties.¹³⁵ Using Robocrystallographer⁷⁵ to convert crystal structures into text, they pretrained a T5 model¹³⁶ with span-masking.¹⁸⁹ LLM-Prop outperformed GNN baselines on MP dataset benchmarks while using 35k fewer training points out of 125k. These studies underscore language models’ capabilities in molecular and crystal representation learning, providing competitive results comparable to those of geometrical GNNs.

3.2. Machine Learning Interatomic Potential

In computational chemistry, the potential energy surface (PES) of a system is of significant importance as it provides valuable information about the system, such as local minima and transition states. In particular, one can sample the PES to collect meaningful points through molecular dynamics (MD) or Monte Carlo simulations. To obtain the PES, one can resort to first-principle methods such as DFT or empirical force fields, such as the Lennard-Jones potential. However, there is a trade-off between accuracy and computational cost, where the former is accurate but slow, while the latter is generally faster but less accurate. Because of this, tasks such as MD simulations, which require numerous energy and force evaluations on large cells, mostly employ empirical force fields at the expense of accuracy. In contrast, AIMD has been performed in limited applications using small cells and short time scales. This trade-off between accuracy and speed poses a challenge to the reliability of MD simulations regarding the accuracy of the potential and the rigor of the simulation setup.

Machine learning interatomic potentials (MLIPs) have emerged as an alternative that offers a compromise between these trade-offs.⁸ MLIPs are predictive models trained on DFT-level total energy and its derivatives, namely forces and stress. Once properly designed and trained, MLIPs serve as both accurate and efficient surrogate potentials, thereby enhancing the reliability of MD simulations. Conventional MLIPs were descriptor-based models, where the descriptors are designed to contain information about the local environment constructed from the combinations of elements in the target system. An intrinsic challenge of this approach is that the trained MLIP cannot be easily transferred to other systems. Moreover, a typical bottleneck in training MLIPs is the generation of data, which often requires an active learning scheme to ensure that the MLIP has seen all relevant configurations likely to be encountered during simulations.

Recently, MLIPs based on GNN architecture have been increasingly reported. The superior performance and descriptor-free nature of GNNs which encode the local environment of any combination of elements into a fixed-length feature vector suggest the potential for a foundational MLIP, which has indeed been reported as a “universal MLIP” (Figure 4).^{31,36,137−139,141,143−145} These universal MLIPs have been trained on large databases encompassing diverse elements, making them transferable across various systems.^190,191 Chen and Ong reported the first universal MLIP, M3GNet, which can perform relaxation or MD simulations of various crystal structures.¹³⁷ They exploited relaxation trajectory data from the MP database, consisting of over 60,000 compounds and 89 elements. Deng et al. developed CHGNet, a universal MLIP capable of predicting the magnetic moment, which is essential in materials such as transition metal oxides.¹³⁸ MACE,⁵⁸ a recent equivariant MLIP, has also been employed to develop universal MLIPs for crystals (MACE-MP-0)³¹ and organic molecules (MACE-OFF23).¹⁴³ In particular, MACE-MP-0 extensively investigated its transferability to a wide variety of systems including solids, liquids, surfaces, porous materials, and even battery cells. Although not always quantitatively accurate, it demonstrated qualitative universality across a wide range at unprecedented levels.³¹

Figure 4.

a. Architecture of GNN-based M3GNet. Reprinted with permission from ref (137). Copyright 2022 Springer Nature. b. Statistics of the MP trajectory data used to train CHGNet. Reprinted from ref (138) under the terms of the CC BY license. c. Data generation pipeline and statistics proposed in MatterSim and e. its zero-shot performance in predicting properties via simulations. Reprinted from ref (36) with permission from the authors. Copyright 2024, the authors. d. Finetuned and zero-shot performances of GNoME in comparison to NequIP⁵⁷ trained from scratch, and f. comparison of the zero-shot performances of universal MLIPs on force prediction of elemental systems. Reprinted from ref (144) under the terms of the CC BY license.

These universal MLIPs not only serve as effective surrogates for tasks such as relaxation, but they also act as foundation models that facilitate the development of MLIPs tailored to specific systems through finetuning when more accurate potentials are required, such as for MD simulations. For example, Jun et al. finetuned CHGNet to study ionic conduction in nitride Li-ion conductors, observing that the anchoring of dopants and Li vacancies lowered the ionic conductivity.¹⁹² Lu et al. finetuned M3GNet to extract Li atomic energies to understand Li-ion migration across interfaces between the cathode and solid electrolytes, where the estimated migration barriers were consistent with the experiments.¹⁹³

Recently, Google reported a universal MLIP called GNoME potential, as part of their work on crystal structure exploration.¹⁴⁴ It was trained on a huge amount of relaxation trajectory data (∼ 10⁸) obtained during the process. Remarkably, through case studies, it was shown that the zero-shot performance of the GNoME potential was comparable to that of SOTA MLIPs trained from scratch (Figure 4d, e). Furthermore, Microsoft reported MatterSim, a universal MLIP trained on a massive dataset of more than 17 million entries, including public databases and in-house datasets calculated over a wide range of conditions covering 0–5000 K and 0–1000 GPa as shown in the top and middle of Figure 4c.³⁶ MatterSim achieved SOTA performance on Matbench Discovery,³⁸ demonstrating its capability of predicting the stability of new, unknown compounds. In addition, it successfully calculated lattice dynamics and thermodynamics, such as phonon dispersion and phase diagrams, respectively, far outperforming the previous universal MLIPs trained solely on relaxation trajectory data (bottom of Figure 4c). These works illustrate the feasibility of a universal foundational MLIP, even with zero-shot or, if necessary, few-shots.

3.3. Inverse Design

Inverse design is a paradigm where desired properties are specified first, guiding the design of new molecules and materials to meet those criteria. High-throughput screening, a common inverse design strategy, generates a large pool of candidates and filters them step by step to identify those with the target properties. However, since initial candidates are often created by fragmenting known molecules and enumerating or substituting them, the success rate tends to be low, and the process relies heavily on the experimenter’s intuition. Generative models, on the other hand, can learn the underlying data structure from large datasets and utilize data distributions to generate novel candidates. Inspired by the success of generative models in producing images^194,195 and text,³² this approach is increasingly being explored in chemistry.

We briefly introduce the concept of the generative model. The goal of the generative model is to learn the probability distribution of data p(x), from which realistic data can be sampled.¹⁹⁶ Training of generative models involves the reconstruction of the training data, which is essentially SSL in nature as aforementioned in Section 3.1.2. For example, a variational autoencoder (VAE) maps by encoder training data x to a feature in latent space enforcing Gaussian distribution, from which latent feature z is sampled and decoded by decoder to the original input x.¹⁹⁷ A more recent approach, the diffusion model, gradually adds Gaussian noise to x until the input data becomes pure Gaussian noise, then removes the noise to reconstruct the x.¹⁹⁸ For the inverse design on a target property, a property predictor can be used in VAE to regularize the latent space by the property,¹⁹⁹ or the denoising process in the diffusion model is conditioned on the property.¹⁶ Another type of generative model, the language model, reconstructs text by either predicting masked words based on the context of a text⁶³ or predicting the next word based on a sequence of previous words.¹⁷⁷

A small foundational generative model for inverse design can be seen as a pretrained generative model which can be adapted to various inverse design tasks. For instance, MatterGen, a diffusion model developed by Microsoft, was first unconditionally pretrained on a large database such as Alexandria¹⁴⁸ and MP,¹³² then finetuned by conditioning on the property such as band gap, elastic modulus, spacegroup, and composition to enable inverse design for each property (Figure 5a, b).¹⁶ Notably, MatterGen significantly outperformed the previous SOTA model by more than a factor of 2 in generating stable, unique, and novel (S.U.N.) materials, which were more than 10 times closer to the DFT local energy minimum (Figure 5c). IBM reported GP-MoLFormer, a transformer pretrained on 1.1 billion SMILES. This was finetuned via a parameter-efficient novel approach called pair-tuning for property-guided optimization, performing comparably to or better than baselines on inverse design for logP, QED (drug-likeness), and DRD2 activity.¹⁴⁹ Gruver et al. proposed finetuning the pretrained LLaMA-2 70B¹⁵² for generation of crystals with desired spacegroup, composition, and E_hull using string representations of crystals and prompting, demonstrating the adaptability of the pretrained LLM to atomistic data.¹⁵¹

Figure 5.

a. Pipeline of diffusion-based MatterGen, b. its performance in inverse design, and c. generating stable, unique, and novel (S.U.N.) materials. Reproduced from ref (16) with permission from the authors. Copyright 2023, the authors.

However, generative models have generally not been explored through pretraining and finetuning schemes; they are most often trained directly from scratch for a target property. As a result, few small foundational generative models exist in this context. Alternatively, some generative models suggest the possibility of a big foundation model, which can be adapted for both property prediction and inverse design. For example, Gómez-Bombarelli et al. first proposed using a VAE in molecular inverse design by integrating it with a property regressor.¹⁹⁹ The VAE successfully optimized molecules for the target property (5 × QED – SAS), while the performance of the regressor was comparable to that of purely supervised GNN. This implies that representations can be learned such that they can be transferred across both property prediction and inverse design. Following this, we will focus on big foundation models that are adaptable to these two domains further in Section 3.5, and we refer the readers interested in generative models in chemistry to refs (18, 196) for more detail.

3.4. Multi-domain: Property Prediction and MLIP

Since MLIPs are predictive models by nature, those that perform well at property prediction are also expected to excel in predicting force fields. Additionally, total energy and forces are related to the global and local features of molecules or crystals, respectively, suggesting that learning force fields itself can serve as an effective pretraining task for property prediction.²⁰⁰ In this section, we review two approaches to utilizing force fields: denoising by learning fictitious force fields, and supervised learning on energy and forces.

3.4.1. Denoising

Denoising was originally developed for generative modeling and aims to learn the derivative of the log probability distribution of data, ∇_x log  p(x), rather than directly modeling p(x).²⁰¹ For configurations of materials, this probability distribution can be expressed by the Boltzmann distribution, p(x) ∝ exp(−E(x)), where E(x) represents the potential energy of configuration x.^153,158 Then, ∇_x log  p(x) ∝∇_xE(x), which is equivalent to the force. In practice, denoising is implemented by predicting noise added to material configurations, often assumed to be Gaussian since equilibrium configurations are located at local minima (Figure 6a). Consequently, denoising can be considered akin to learning the virtual forces that guide noisy coordinates back to their original equilibrium positions.¹⁵⁸

Figure 6.

a. Illustration of denoising as a pretraining for properties and force field prediction. Reprinted from ref (158) with permission from the authors. Copyright 2023, the authors. b. Supervised pretraining on force fields dataset via joint multi-domain pretraining (JMP) proposed by Shoghi et al. Reprinted from ref (200) with permission from the authors. Copyright 2024, the authors.

This denoising approach may benefit downstream tasks such as the prediction of forces and other quantum mechanical properties that are highly sensitive to 3D coordinates.^{37,153,156,158,161,184,202−204} For instance, Jiao et al. achieved a notable improvement in force prediction on the MD17 dataset, reducing the average error from 0.2086 to 0.0968 kcal/ after pretraining, and outperformed other SSL methods across 11 out of 12 properties in the QM9 dataset.¹⁵⁸ Denoising has also been explored for predicting properties of crystalline materials.^184,203,204 Notably, Song et al. applied denoising to fractional coordinates and lattice parameters using a denoising diffusion probabilistic model (DDPM)¹⁹⁸ framework.¹⁸⁴ This method achieved SOTA performance on the JARVIS-DFT benchmark and showed better results in data-limited scenarios, such as with experimental data.

3.4.2. Supervised Pretraining

Thanks to advances in computational resources and speed, high-quality force field databases are expanding and becoming readily available. As a result, energy and force data can serve as effective pretraining labels, both in terms of abundance and relevance to downstream tasks.^146,200,205 Gao et al. leveraged 86 million DFT-calculated energies from the PubChem PM6²⁰⁶ database for pretraining to adapt for force prediction and molecular property tasks.²⁰⁵ Importantly, they applied zero-force regularization, given that the molecular structures in the training data were optimized. Feng et al. introduced a force-centric pretraining strategy combining denoising and supervised learning for equilibrium and nonequilibrium conformations, respectively.³⁷ This approach not only improved test set performance for force prediction but also enhanced stability in MD simulations and accurately reproducing interatomic distance distributions, demonstrating its practicality for MLIP. Additionally, Jia et al. demonstrated that explicitly learning the derivatives of total energy, i.e. forces and stress, generally results in superior performance across a variety of downstream tasks compared to training solely on energies or using denoising methods.²⁰⁷

Recently, Meta’s FAIR team introduced a novel joint multi-domain pretraining (JMP) strategy, in which the model is trained simultaneously on diverse energy and force databases (Figure 6b).²⁰⁰ To handle the inconsistencies in DFT calculation setups and differences in database sizes and system scales, they normalized the data and modified commonly used loss functions. This approach achieved SOTA performance in 34 out of 40 downstream tasks, spanning various databases and including properties and forces of molecules, crystals, and MOFs, underscoring the effectiveness of their pretraining method. The universal MLIP discussed in Section 3.2 can also be adapted for property prediction tasks. Indeed, Chen et al. and Yang et al. demonstrated that each universal MLIP’s performance as a property predictor was comparable to or exceeded SOTA results.^36,137 These studies highlight the efficacy of learning force fields as a pretext task to learn the generic representations adaptable to a wide range of property predictions.

3.5. Multi-domain: Property Prediction and Inverse Design

While the property-directed inverse design may suggest novel materials beyond the known chemical space, it remains essential to evaluate the generated data using property prediction tools. In this sense, a big foundation model that is adaptable to both property prediction and inverse design would play a key role in materials discovery. This can be achieved by learning representations that are transferable across both domains, as discussed in Section 3.3. In this section, we focus on approaches to big foundation models, categorized into two areas: multimodal learning and leveraging LLMs.

3.5.1. Multimodal Learning

One of the earliest multimodal models is KV-PLM, which leveraged two modalities: molecular structure and text description (Figure 7a).¹⁶² Specifically, molecular structure was represented using SMILES, tokenized with the byte pair encoding (BPE)²⁰⁸ algorithm to capture frequent substring patterns in a molecule, while text descriptions were obtained from S2orc,¹⁶³ an academic paper corpus. The segmented SMILES tokens were inserted into the text, which was used to pretrain BERT as a backbone model via masked language modeling. The model successfully generated drug molecules from the input text and performed comparably to baselines such as D-MPNN,²⁰⁹ showcasing the potential of language-based foundation models. MolT5, proposed by Edwards et al., took a similar approach, leveraging both text and SMILES to train the T5¹³⁶ model, with text-based inverse design and molecule captioning (rather than property prediction) investigated as downstream tasks.²¹⁰

Figure 7.

Multimodal foundation models. a. Schematic of multimodal KV-PLM, using SMILES and text by fusing them into unified data. Reprinted from ref (162) under the terms of the CC BY license. b. Multimodal model proposed by Liu et al., jointly training the structure and text encoders for each modality. Reprinted with permission from ref (40). Copyright 2023 Springer Nature. c. Workflow of SPMM proposed by Chang and Ye, which aims to learn joint representation of structure and properties. Reprinted from ref (41) under the terms of the CC BY license. d. Finetuning the public ChatGPT proposed by Jablonka et al. Reprinted from ref (175) under the terms of the CC BY license.

Su et al. employed molecular graphs instead of SMILES as a structural modality, with text description as another modality.¹⁶⁵ A pretrained BERT model^162,166 and a GNN model from GraphCL¹⁰⁵ were used as backbone text and graph encoders for their multimodal model, respectively, which was then trained using contrastive learning between the text and graph modalities. For inverse molecular design, they utilized an external model, MoFlow,²¹¹ which transforms q sampled from the Gaussian distribution into molecular graphs. They optimized q to maximize the cosine similarity between the input text and generated molecular graph representations, successfully generating valid molecules based on descriptions at various levels. In property prediction, the graph encoder was finetuned, outperforming all compared SSL methods on average in the MoleculeNet benchmark.

Liu et al. proposed a similar multimodal learning framework, MoleculeSTM, which can be adapted to structure-text retrieval, text-based molecule editing, and property prediction (Figure 7b).⁴⁰ For the zero-shot molecule editing, another generative model is employed, by optimizing representations to maximize similarities to both representations encoded by the pretrained text encoder and the generative model. In the 4 editing tasks including single- and multi-objective editing, binding-affinity-based editing, and drug-relevance editing, MoleculeSTM achieved superior hit ratios compared to baselines including genetic search. Luo et al. proposed MolFM, where a knowledge graph was incorporated as regularization in addition to molecular graph and text encoders.¹⁶⁷ KV-PLM¹⁶² and GraphMVP¹¹¹ were employed to initialize their encoders, with four pretraining objectives consisting of structure-text contrastive loss, cross-modal matching loss, masked language modeling loss, and knowledge graph embedding loss. In the downstream inverse design, the MolT5²¹⁰ decoder was employed to generate SMILES from the input text.

More recently, Chang and Ye proposed SPMM to learn the joint representation of structure and property by embedding SMILES and a property vector (PV) in a common embedding space, enabling bidirectional PV-to-SMILES generation and property prediction (Figure 7c).⁴¹ PV consisted of 53 molecular properties calculated by RDKit and was treated as a 53-length sentence. Notably, PV was randomly masked to allow the use of partial properties in practice. SMILES generation takes up to 53 properties and requires no additional training, successfully generating valid and novel molecules with desired PVs in all tested cases, including scenarios with all 53 properties, a single property, four properties, and no properties provided. In addition, SPMM can predict the PVs of input SMILES without finetuning but can be finetuned for enhanced property prediction, achieving performance comparable to the SOTA on the MoleculeNet benchmark. These studies demonstrate that multimodal learning is a promising approach for developing big foundation models in chemistry, enabling a range of downstream tasks across domains and the use of more specific conditions to design the desired molecules.

3.5.2. Large Language Model

A large language model such as ChatGPT is renowned for its human-like performance and versatility in understanding context and generating text, making it one of the most popular foundation models. This has motivated researchers to leverage LLMs to solve scientific problems, leading to the development of science-specific LLMs trained on various scientific corpora.^{166,212−214} Furthermore, more specialized LLMs have been developed that are adaptable to specific chemical problems, including text-based generation, property prediction, molecule captioning, molecule retrieval, reaction prediction, and more.^{33,34,215,216}

Zhao et al. proposed a Dialogue Foundation Model for Chemistry (ChemDFM),¹⁷² built upon the pretrained LLaMa-13B.¹⁷³ They trained LLaMa-13B on 3.8M research articles and 1.4K chemistry textbooks, followed by instruction tuning to better adapt the model to chemical languages, including molecular notations, using 2.7M instructions from chemical databases. In property prediction and inverse design tasks, the model performed better than 10-shot generalist LLMs such as GPT-4 and was comparable to conventional specialist models. Livne et al. introduced nach0, pretrained on both NLP and chemical domain datasets.¹⁷⁴ nach0 was finetuned with instruction tuning in a multi-task manner for cross-domain tasks such as text-guided molecule design and property prediction, outperforming SOTA baselines. Notably, these models are capable of interacting with humans as they were pretrained on large corpora, distinguishing them from the aforementioned multimodal language models pretrained purely on SMILES and text descriptions.

Instead of pretraining LLMs manually, researchers can leverage publicly available pretrained LLMs and adapt them for chemistry. Jablonka et al. proposed finetuning GPT-3 via the OpenAI API and benchmarked the model for property prediction and inverse design (Figure 7d).¹⁷⁵ The fine-tuned GPT-3 performed comparably to or even outperformed conventional specialized models in classification, particularly in low-data regimes. In a case study on the inverse design of molecules for photoswitches, the finetuned GPT-3 was able to generate valid molecules with higher novelty using a higher softmax temperature, albeit with an increased risk of invalidity. Choudhary employed GPT-2 and Mistral 7B¹⁷⁸ for property prediction and inverse design of crystals, respectively,¹⁷⁶ using text descriptions generated by ChemNLP.⁷⁶ These studies suggest a convenient approach to leveraging foundation models for materials discovery by simply finetuning public LLMs.

4. Trends and Future Directions

4.1. Scope

The scope of foundation models can be categorized into three perspectives. The first is the type of materials the foundation models can handle, such as molecules, crystals, surfaces, MOFs, and so on. A large portion of reported foundation models have targeted molecules, and most others have focused on a single type of material. Meanwhile, a foundation model can leverage shared chemistry across different materials, hopefully allowing for complementary learning, especially for those with relatively limited data. This would enhance flexibility across diverse material types. Supervised learning of this concept was explored by Shoghi et al.,²⁰⁰ but a similar approach could be promising in SSL with careful integration and utilization of different databases across a wide range of materials.

Second is the modality of data. Popular modalities for molecules include SMILES, graphs, text descriptions, and properties. These modalities complement each other and offer opportunities for more diverse tasks. For example, as 3D geometric graphs provide richer structural information than 2D graphs and SMILES, leveraging 3D information can aid downstream tasks where only 2D graphs or SMILES are available.¹¹¹ In this sense, exploring additional modalities, such as images and spectral data, and investigating effective approaches to maximize their utility is a promising direction.⁹² For example, Suzuki et al. utilized X-ray diffraction (XRD) patterns as a modality alongside crystal graphs to learn the structure–functionality relationship.²¹⁷ However, multimodal learning for crystals remains underexplored, warranting further investigation.²¹⁸

Finally, the downstream tasks. The ultimate goal of a foundation model in chemistry is to maximize versatility across downstream tasks, regardless of domain. Progress in this direction is thought to depend on the previous two challenges, as each material type and modality constitutes a task in itself (e.g., predicting a molecule’s properties from its SMILES). It is more effective to adapt a foundation model to a learned material type and modality, or at least related ones. Indeed, multimodal learning has proven to be one of the most promising approaches toward a big foundation model across domains, as discussed in Section 3.5.1. Furthermore, it is reasonable to expect that a foundation model trained on vast, diverse datasets could adapt well to new chemistry and tasks, an emergent capability.²⁹ Consequently, we anticipate that discovering effective methods to extend the range of materials and modalities will be key to developing big foundational models capable of encompassing any domain.

4.2. Performance

Central to the success of foundation models is scale, both in data and model size.²⁹ Hence, understanding the scalability of data and models in chemistry is crucial. It should be noted that although the high cost of labeling data makes SSL popular for pretraining, it is not straightforward to obtain large meaningful unlabeled data efficiently. Naively generating molecules and crystals risks producing a large amount of invalid and unstable material data. While generative models can successfully produce realistic data, they generally fill gaps within known chemical space rather than expand it, as they are trained on existing databases. Therefore, advanced techniques are needed to explore the configuration space and discover meaningful material structures beyond what is currently known, often requiring extensive domain expertise.²¹⁹ Nevertheless, unstable material data may still be useful to some extent, for example, in learning symmetries of crystals, which are important for determining their properties but challenging to capture.^35,220

The integration of existing databases is one of the most straightforward ways to enlarge the dataset for training large-scale foundation models. However, data acquisition method must be compatible between existing databases to ensure seamless integration, which is often not the case, necessitating a proper approach to address inconsistencies.²⁰⁰ For example, Shiota et al. recently proposed a total energy alignment method requiring minimal recalculations to integrate databases calculated using different ab initio packages (e.g., MP¹³² using the Vienna Ab initio Simulation Package (VASP)²²² and OFF23¹⁴³ using Psi4²²³).²²¹ The importance of consistent data is further highlighted by Pengmei et al., who found that noisy labels, such as DFT-calculated HOMO–LUMO gaps with high uncertainty depending on the choice of DFT functionals and basis sets, could limit scalability and OOD performance.²²⁴ In conclusion, both the quantity and quality of data are crucial for the effective scaling and performance of foundation models.

As with large amounts of data, it is important to design scalable models that can harness this information to learn meaningful knowledge. For example, transformers can scale up to tens or hundreds of billions of parameters while improving performance.³² In this regard, language models for chemistry represent a promising architecture that has already been extensively studied. However, the scalability of GNNs has rarely been explored, with complex equivariant models typically having fewer than a million parameters.³⁸ Simply increasing the dimensionality and message-passing layers is not always beneficial due to the curse of dimensionality and oversmoothing, the latter of which Godwin et al. proposed mitigating by introducing noisy nodes.²²⁵ Very recently, Sypetkowski et al. demonstrated that GNNs could scale up to 3 billion parameters with consistent improvements in molecular property prediction and highlighted the importance of model width in driving finetuning performance.²²⁶ In another recent study, Pengmei et al. found distinct scaling characteristics during pretraining compared to other fields like language modeling.²²⁴ Such studies should be conducted more actively to advance large-scale GNNs, paving the way for developing foundation models in chemistry.²²⁴

A common observation in many studies suggests that exploiting both local and global information about materials is effective in learning good representations. This is because properties may depend on local, global, or both types of information, providing such representations with maximum versatility in various downstream tasks involving properties. This can be achieved through contrastive learning between local and global views, predictive learning on local motifs and global properties, or generative learning to reconstruct masked nodes and entire graphs. Contrastive learning with data augmentation has also proven effective with carefully designed augmentation methods at both the node and graph levels. Additionally, supervised learning on force fields allows the model to effectively learn both local and global aspects, each represented by atomic forces and total energy. Overall, pretraining tasks that focus on both aspects represent a promising approach deserving further exploration.

While many reported foundation models demonstrate excellent performance compared to conventional approaches on benchmarks such as MoleculeNet,¹⁸² Matbench,⁹⁴ and JARVIS-DFT,¹⁴⁰ more attention should be paid to their performance in the limited data regime, a typical challenge in chemistry and materials science.³⁵ For example, experimental datasets such as OQMD-EXP^131,184 and high-fidelity quantum mechanical calculation data (e.g., coupled cluster)^160,227,228 of relatively smaller amounts can serve as useful benchmarks. Performance tests on varying training data sizes would also provide insights into the scaling characteristics of the models.^153,184,202 Additionally, robust extrapolation capability is another key factor in chemistry that requires more focus.³⁷ In this regard, constructing benchmark datasets across various domains can help objectively assess a model’s OOD performance. For example, Fu et al. reported a benchmark suite and metrics to evaluate the robustness of MLIPs during MD simulations, which is not necessarily guaranteed by low test set prediction error.²²⁹ The Matbench Discovery also serves as a benchmark to assess the relaxation performance of universal MLIPs on unseen structures.³⁸ These benchmark datasets would aid in validating foundation MLIPs in more practical settings.

4.3. Efficiency

Since the computational burden increases as the size of models grows, efforts have been made to improve model efficiency, particularly for LLMs.²³⁰ For example, quantization reduces the number of bits required to represent model weights, pruning aims to identify and remove redundant weights and components, and matrix decomposition alleviates computational overhead in matrix multiplication. While these techniques are model-agnostic in principle, thorough experiments on GNNs for chemistry are necessary. Meanwhile, Sourek et al. proposed a technique to compress GNNs without loss of information by capturing symmetries, i.e., shared correlations in the data, taking advantage of the structured convolutional nature of GNNs.²³¹ Additionally, the necessity of equivariance in 3D geometric GNNs which incur large computational costs may not always manifest, requiring further investigation.²³² Studies on model compression tailored to GNNs would represent a promising direction for developing efficient foundation models for chemistry.

Efficiency is particularly critical for MLIPs, as large-scale simulations involving millions of timesteps are expected in MD simulations driven by MLIPs. However, foundation models can be very expensive even in the inference stage, limiting their utility as MLIPs. Knowledge distillation²³³ may offer a solution, where knowledge from an expensive but accurate and generalized model (the teacher model) is distilled into a lighter and more efficient model (the student model).²³³ Ekström Kelvinius et al. first proposed using knowledge distillation in GNN-based MLIPs by aligning embeddings obtained from the teacher and student models.²³⁴ Gong et al. introduced ensemble distillation, achieved by training a single MLIP on snapshots sampled from MD simulations labeled by an ensemble of MLIPs.²³⁵ While knowledge distillation has primarily been studied in classification models and rarely addressed in regression tasks, exploring what constitutes “knowledge” could unveil significant opportunities. For example, while the decomposition of total energy into atomic energies in the MLIP formalism is not unique and can be somewhat ad hoc if ill-trained, they can be correctly predicted in a well-generalized model, reflecting physical and chemical atomic interactions,^236−238 potentially serving as knowledge for MLIPs.

4.4. Interpretability

As many researchers rely on the use of foundation models, the limitations of foundation models could be easily ignored. As an example, some foundation models are still vulnerable to generating hallucination, generating plausible yet nonfactual content,²³⁹ and the model collapsed²⁴⁰ outputs, which lack sample diversity. Thus, when we use a foundation model, it is important to verify that the generated outputs have valid and safe meaning in chemistry domain. Especially, when the generated contents are biased to a narrow distribution of training data, the novel discovery of chemical compounds or properties is limited. To solve these problems, it is recommended to utilize the model interpretability techniques which explain sample generation mechanisms in the foundation models.²⁴¹ For example, interpretable features from a foundation model, Claude 3, were extracted and visualized.²⁴² Moreover, it was found that specific concepts learned in foundation models can be localized and edited at the word level.^243,244 Recently, a method to accurately identify source documents used to train foundation models was proposed.²⁴⁵ Such advances in model interpretability make it possible to better understand foundation models,²⁴⁶ paving the way for their more reliable use in chemistry.

5. Conclusion

A foundation model in chemistry is expected to solve various problems in the field, guiding researchers to achieve the long-standing goal of materials discovery and understanding the behavior of matter. The current status of foundation models in chemistry is still premature, and many challenges must be overcome for these models to be implemented at a practical level. However, there are clear trends in effective pretraining approaches and learning schemes, and importantly, databases are expanding. These offer promising opportunities for foundation models in chemistry in the future, toward new possibilities beyond the domains focused in this perspective.

The authors declare no competing financial interest.