Machine Learning Methods for Small Data Challenges in Molecular Science

Abstract

Small data are often used in scientific and engineering research due to the presence of various constraints, such as time, cost, ethics, privacy, security, and technical limitations in data acquisition. However, big data have been the focus for the past decade, small data and their challenges have received little attention, even though they are technically more severe in machine learning (ML) and deep learning (DL) studies. Overall, the small data challenge is often compounded by issues, such as data diversity, imputation, noise, imbalance, and high-dimensionality. Fortunately, the current big data era is characterized by technological breakthroughs in ML, DL, and artificial intelligence (AI), which enable data-driven scientific discovery, and many advanced ML and DL technologies developed for big data have inadvertently provided solutions for small data problems. As a result, significant progress has been made in ML and DL for small data challenges in the past decade. In this review, we summarize and analyze several emerging potential solutions to small data challenges in molecular science, including chemical and biological sciences. We review both basic machine learning algorithms, such as linear regression, logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), kernel learning (KL), random forest (RF), and gradient boosting trees (GBT), and more advanced techniques, including artificial neural network (ANN), convolutional neural network (CNN), U-Net, graph neural network (GNN), Generative Adversarial Network (GAN), long short-term memory (LSTM), autoencoder, transformer, transfer learning, active learning, graph-based semi-supervised learning, combining deep learning with traditional machine learning, and physical model-based data augmentation. We also briefly discuss the latest advances in these methods. Finally, we conclude the survey with a discussion of promising trends in small data challenges in molecular science.

Graphical Abstract

1. INTRODUCTION

In recent years, machine learning (ML), including deep learning (DL), has made remarkable advancements in a wide range of research fields, including science, engineering, technology, medicine, and industry,^1–4 marking a significant milestone in data-driven discovery. Sophisticated algorithms, such as graph convolutional networks (GCNs),⁵ convolutional neural networks (CNNs),⁶ recurrent neural networks (RNNs),⁷ and Generative Adversarial Networks (GANs),⁸ are aided by powerful computing resources, such as graphics processing units (GPUs), to achieve success in ML and DL. The main reason behind these achievements is the ability to accurately estimate the behavior in unknown domains by quantitatively learning patterns from a sufficient number of training samples. However, in scientific fields, it is often challenging to obtain large labeled training samples due to various restrictions or limitations such as privacy, security, ethics, high cost, and time constraints. Fields such as computer vision,⁹ language translation,¹⁰ speech recognition,¹¹ and game playing¹² may have large-scale data sets with billions or even trillions of data points, but this is typically not the case in scientific research. For example, in drug discovery,^13,14 the discovery of properties of new molecules to identify useful ones as new drugs is constrained by toxicity, potency, side effect, partition coefficient (log P), solubility (log S), and various other pharmacokinetics and pharmacodynamics metrics. As a result, there are few records of successful clinical candidates for a given target. When the number of training samples is very small, the ability of ML-based or DL-based models to learn from observed data sharply decreases, resulting in poor predictive performance. Therefore, it is very important for the scientific community to learn and generalize effectively the data from very few training samples.

Efficiently learning from very few training samples holds great theoretical and practical significance in the fields of ML and DL. First, it can help avoid the prohibitively high cost of acquiring data and performing costly annotations in certain data-intensive applications. Second, it can enable the construction of a low-cost and speedy model for an emerging task that only has a few temporarily available samples, which can illuminate potential laws earlier in the exploration process. Driven by these promising advantages and the practical need for affordable learning, learning from very few training samples has become a popular research topic. However, despite related ML approaches such as small or one-sample learning, zero-shot learning,¹⁵ one-shot,¹⁶ or few-shot learning,^17,18 the research progress on this problem has been slower in the past decade compared to that of large sample learning, due to its intrinsic difficulty. For instance, if a learning algorithm is executed on a task with very few training samples using just vanilla learning techniques without any advanced learning strategies or specific model design, serious overfitting may occur, significantly reducing the predictive power of the model.¹⁹

Overall, there are several viable strategies to improve the predictive power of ML or DL models when dealing with small scientific data sets. Commonly used strategies include transfer learning,^20,21 combining DL and ML,^22,23 GANs,^24,25 variational autoencoder (VAE),^26,27 self-supervised learning (SSL),^28,29 long short-term memory (LSTM),^30,31 data augmentation based on physical models,^32,33 active learning (AL),^34,35 and semi-supervised learning.^36,37 However, no paper has provided an organized taxonomy linking these techniques. Therefore, in this review, we conduct a survey on ML or DL prediction using small scientific data sets and aim to create a taxonomy that connects these techniques.

The remaining sections of this paper are organized as follows. ML preliminaries are presented in section 2. Section 3 provides a brief overview of several of the main methods used for dealing with data scarcity. Section 3.11 details the theory of transfer learning and its applications in the context of small data sets. Sections 4.1 and 3.7 discuss the methods of combining DL with traditional learning and those based on GANs, respectively. In section 3.8, we outline VAE-based methods for dealing with small training set sizes. Section 3.9 surveys the approach of SSL to small data sets. Sections 3.6 and 3.12 cover LSTM techniques and AL methods, respectively. In sections 3.13 and 4.2, we delineate the Merriman–Bence–Osher method and physical model-based data enlargement, respectively. Section 4 discusses several perspectives for dealing with small data challenges in molecular science, including combining DL with traditional ML, physical model-based data augmentation, natural language processing (NLP), and generative networks. Finally, section 5 offers an outlook on future developments.

2. MACHINE LEARNING PRELIMINARIES

2.1. Supervised, Unsupervised, Semi-Supervised, and Self-Supervised Learning Strategies

In supervised learning,^38,39 a data set containing input and output pairs is used to train a function that maps feature vectors (input) to labels (output). The data set is split into a training set and a test set, with the former used to adjust the model parameters for accurately predicting the outputs for the input examples in the training set. After the model is trained, its generalization ability is evaluated by testing its performance in the test set.

Supervised learning encompasses various types, including classification, regression, naïve Bayes (NB) models, ⁴⁰ random forest (RF) models,⁴¹ support vector machines (SVM),⁴² and neural networks (NNs),⁸ among others. These algorithms have found widespread application in biological and chemical fields. For instance, Lazarovits et al.⁴³ constructed NN models to investigate the mechanism of liver and spleen uptake by nanoparticles, finding that it was due to protein adsorption on their surfaces. Sandfort et al.⁴⁴ concatenated 24 fingerprint representations into a 71 375 dimensional vector, which was then used for various supervised learning tasks related to chemical reactivity. Additionally, there is a growing interest in applying supervised learning techniques to predict drug side effects.⁴⁵ Munoz et al.⁴⁶ used different models, including logistic regression, RF, decision trees, and others, to predict the side effects of biological molecules. Similarly, Zhou et al.⁴⁷ applied boosted RF classifiers to predict the side effects of protein targets, therapeutics, transport proteins, enzymes, pathways, and chemical structures of drugs.

In unsupervised learning, the available data set does not have labeled training examples, and the objective is to uncover patterns or relationships in the data.^48,49 One of the most common types of unsupervised learning are clustering, which involves grouping unlabeled data points based on their similarities and differences. This process aims to group data points into clusters in such a way that those in the same group have the highest similarity to each other while points in different groups have little or no similarities. Another type of unsupervised learning is data compression or dimensionality reduction,⁵⁰ which aims to represent high-dimensional data in a lower-dimensional space while preserving as much information as possible. This technique can significantly reduce computing or storage costs while making the ML model run much faster. Unsupervised learning has become a crucial tool for handling the increasing amount of data generated by atomic and molecular simulations in biochemistry. Glielmo et al.⁵¹ provided a discussion of the latest algorithms for feature representation of molecular systems used in downscaling and clustering models. Basdogan et al.⁵² employed a nonlinear dimensionality reduction algorithm to create a two-dimensional visual representation of the similarity between solute environments in microsolvation clusters of different sizes.

Semi-supervised learning⁵³ involves a data set that includes both labeled and unlabeled examples. The objective is to learn a function that can predict the labels for the unlabeled samples using the labeled ones. Semi-supervised learning can be used for various tasks, such as classification, regression, clustering, and association, and offers the benefit of reducing expenses on manual annotation and data preparation time. There are several approaches to semi-supervised learning, including self-training,⁵⁴ cotraining,⁵⁵ and multiview learning,⁵⁶ with the selection of a specific approach depending on the data set’s properties and the task requirements. MicroRNAs are noncoding RNAs closely associated with many human diseases in the biomedical field. Ji et al.⁵⁷ treated MicroRNAs disease association prediction as a semi-supervised learning problem and proposed a novel method to predict potential MicroRNA-disease associations, a new method for predicting potential MicroRNAs-disease associations. In healthcare, Yin et al.⁵⁸ introduced deep forest and semi-supervised self-training to address disease classification and gene selection for different types of diseases. Experimental results demonstrated that the proposed model could achieve good results in both disease classification and causative gene identification.

SSL, also known as predictive or pretext learning, is an ML process where a model trains itself to learn one part of the input from another part of the input.⁵⁹ The goal is to learn useful representations from unlabeled data that can help with downstream learning tasks such as classification or object detection. In SSL, the unsupervised problem is transformed into a supervised problem by autogenerating the labels. To make use of the huge quantity of unlabeled data, it is crucial to set the right learning objectives to get supervision from the data itself. Hence, SSL is particularly useful for tasks where it is difficult to obtain labeled data or where the amount of labeled data is limited. Some common SSL tasks include predicting missing elements in data,⁶⁰ reconstructing data from corruptions or perturbations,⁶¹ and so on. SSL has recently achieved tremendous success in the fields of biology and chemistry. Wang et al.⁶² proposed a cloze-style SSL model, MolCloze, in 2021 to obtain a generic information representation for molecular property prediction tasks. In 2022, Zhang et al.⁶³ introduced the concept of SSL to develop HelixADMET, a robust and end point scalable absorption, distribution, metabolism, excretion, and toxicity (ADMET) system. HelixADMET generated a pretrained model to efficiently screen out unwanted drug candidates in the early stages of drug discovery.

2.2. Regression, Classification, Clustering, and Dimensionality Reduction Tasks

Regression ML is used to understand the relationship between dependent and independent variables and commonly predicts a continuous value based on the input variables. The main goal of regression problems is to estimate a mapping function based on the input and output variables. There are various types of regression algorithms that are widely used in the biochemical domain, including linear regression,⁶⁴ decision tree regression,⁶⁵ principal components regression,⁶⁶ RF regression,⁶⁷ support vector regression,⁶⁸ and polynomial regression.⁶⁹ For example, in computer-aided drug design, multiple linear regression models are often used for pattern recognition, structural similarity, and binding energy prediction to screen promising drug candidates for COVID-19 therapy and quantitative structure–activity relationships when assessing the structural stability and densification of drugs in complex with the major protease of SARS-CoV-2.⁷⁰ Yan et al.⁷¹ used multiple linear regression model and SVM methods to predict the inhibitory activity of 117 Aurora-A kinase inhibitors, respectively. Additionally, Ye et al.⁷² applied the established molecular docking-based SVM regression model to the design of new NF-B-inducing kinase inhibitors.

Classification predictive modeling involves approximation of a mapping function from input variables to discrete output variables, which are typically labeled or categorized. The mapping function is used to predict the class or category for a given observation. In many cases, classification algorithms predict a continuous value as the probability of an example belonging to each output class. These probabilities can be interpreted as the likelihood or confidence of an example belonging to a particular class. Common classification algorithms include linear classifiers, SVMs, decision tree classification, k-nearest neighbor (KNN), and RF classification, among others.^40,73,74 There are numerous applications of these methods in biology and chemistry. For instance, Arian et al.⁷⁵ utilized the KNN algorithm to distinguish between active and inactive protein kinase inhibitors and evaluated the performance of the model using SVM and NB classification methods.

Clustering or cluster analysis^76,77 is the task of grouping a set of objects into homogeneous groups or clusters while ensuring that objects in different groups are dissimilar. Clustering can be considered an unsupervised task, as it aims to describe the hidden structure of the objects. Each object is described by a set of features. The key step in dividing objects into clusters is to define the similarity or distance between the different objects. There are many clustering algorithms, including hierarchical clustering,⁷⁸ centroid-based clustering,⁷⁹ distribution-based clustering,⁸⁰ density-based clustering,⁸¹ and grid-based clustering.⁸² In order to improve the classification of primary breast cancer and identify disease subgroups relevant to patient management, Ferro et al.^83,84 used four different clustering methods. Their findings showed that applying unsupervised learning to primary breast cancer data was a promising approach to enhance the classification of primary breast cancer and define subclasses of treated patients.

Dimensionality reduction⁸⁵ is the process of reducing the number of random variables in a data set while retaining as much relevant information as possible. The goal of dimensionality reduction is to transform high-dimensional data into data of lower dimensions, making them easier to analyze, visualize, and understand. Dimensionality reduction is commonly used as a preprocessing step before supervised learning and to remove noise in the data. There are several common dimensionality reduction methods, including principal component analysis,⁸⁶ factor analysis,⁸⁷ t-distributed stochastic neighbor embedding (t-SNE),⁸⁸ the uniform manifold approximation and projection,⁸⁹ and residue-similarity scores,⁹⁰ among others. Several applications of biochemistry have utilized ML techniques. For example, to investigate the structure and binding interactions of HIV-1 protease and P2 ligands, Karnati et al.⁹¹ performed principal component analysis to identify differences in conformational changes induced by inhibitor binding. In 2021, Bort et al.⁹² used t-SNE to explore the structure of bioactive organic molecule data sets.

3. METHODS FOR SMALL MOLECULAR DATA CHALLENGES

3.1. Basic Machine Learning Algorithms

Since Arthur Samuel proposed the concept of ML in 1956,⁹³ a large number of algorithms have been developed. Some of the most traditional algorithms include KNN, which was proposed by COVER in 1968,⁹⁴ SVM, which was introduced in 1995 by Cortes,⁹⁵ and RF which was also proposed in 1995 by Ho.⁹⁶ These classic and fundamental algorithms have been widely applied in various fields such as data mining,^97–99 statistical learning,^100,101 and computer vision.^102,103 ML is becoming increasingly popular in the fields of biological medicine and chemistry, particularly in healthcare and COVID-19 research.^104–106 This is due to its potential to aid in disease diagnosis,¹⁰⁷ data pattern detection,^108,109 patient management,¹¹⁰ and other areas. In this context, basic ML algorithms will be introduced in the field of small molecules. Special algorithms like CNNs and artificial neural networks (ANN) will be discussed in detail later.

In the field of drug–target interaction, traditional chemical experiments can be both expensive and time-consuming. Although many methods based on different principles have been developed to measure the similarity of drugs or targets, their results are often unsatisfactory. In 2018, Chen et al. proposed a new method for identifying drug–target interaction using the Gradient Boosted Decision Trees (GBDT) ML algorithm.⁴⁰ This method combined drug and protein identifiers, descriptors, and negative information to predict drug–target interactions. The data set used in the experiment consisted of 4950 drugs and 2313 human protein interactions, with 609 drug characteristics and 1819 protein characteristics. The GBDT algorithm was compared to six other methods in the experiment, and the results are shown in Table 1. The experimental results indicated that the GBDT algorithm outperformed other advanced methods, particularly when the data set was small.

Table 1.

Prediction Results Based on Different ML Methods for Identifying Drug–Target Interactions^a

Method	AUC	Accuracy
Naive Bayes	0.54285	0.445622
neural net	0.55611	0.544142
SVM	0.56119	0.597514
logistic regression	0.62449	0.619996
nearest neighbors	0.71011	0.663864
random torest	0.87473	0.817584
our approach	0.91095	0.871931

^aReproduced with permission from ref 40. Copyright 2018 Springer Nature.

Drug-induced toxicity is a significant side effect that requires consideration during drug development. However, current experimental methods used to evaluate drug-induced toxicity are often time-consuming and expensive, which make them unsuitable for large-scale assessments during the early stages of drug discovery. In 2014, Zhou et al. proposed a computational prediction model of drug-induced toxicity based on SVM.¹¹¹ The study included 572 samples from a small toxicity data set, and to compare the performance of the proposed model, the researchers applied NB¹¹² and recursive partitioning¹¹³ methods to the same data set. Among all the prediction models, drug-induced toxicity based on SVM achieved the best performance, with prediction accuracies of 85.33% and 83.05% for the two independent test sets, respectively.¹¹¹ In comparison, the Bayesian model yielded prediction accuracies of 76.09% and 74.58% in the two independent test sets, while the recursive partitioning model resulted in prediction accuracies of 79.89% and 77.97% in the same two independent test sets. Based on these experimental results, the drug-induced toxicity based on the SVM model outperformed the other two models.

There is growing interest in the application of ML and DL across the life sciences, including drug discovery. In 2022, Siemers et al. identified the minimal data requirements for learning with activity-based composite classification, which serves as an example application.¹¹⁴ ML binary classification models were constructed using increasingly larger training sets, starting from a minimal set that included only one active compound (and two inactive compounds) and extending up to a training set containing 600 active compounds (and 1200 inactive compounds), which was used for model construction. In chemical informatics, message-passing neural networks are increasingly used for the deep representational learning of molecular diagrams and the prediction of molecular properties. As a control, a simple KNN classifier was used. To explore the sustained high performance of the KNN classifier, the researchers systematically extracted simulated sequences from 20 randomly selected activity classes using the composite core relation algorithm. The calculation protocol is shown in Figure 1. The results of the control calculations showed an increasing size of the training set consisting of the same number of active and inactive instances.¹¹⁴ This work has the potential to impact the prospective use of prediction models, particularly for emerging targets where there are typically limited compound data.

Figure 1.

Calculation protocol for molecular classification. For each activity class, eight independent trials with different seeds were carried out. For each trial, a test data set was randomly chosen containing 100 active and 100 inactive compounds. For each training set size, training and validation data sets were assembled. Reproduced with permission from ref ¹¹⁴. Copyright 2022 Elsevier.

MicroRNAs play a crucial role in many pathological processes by inhibiting translation through interaction with specific target mRNAs. These miRNAs can act as either oncogenes or tumor suppressors. In 2022, Yu et al. used a variety of ML algorithms to build models predicting up–down and up–down pairs on the training parts of data set1 and data set2, respectively.¹¹⁵ The flowchart of this study is illustrated in Figure 2. A model was constructed using data set1 (2096 positive pairs and 2096 manually constructed negative pairs) to predict upregulated pairs of small molecules and MicroRNAs. Similarly, data set2, with 1591 positive and 1591 negative pairs, was used to build a model for predicting down-regulated pairs. The RF algorithm showed the best performance. On the test data set, the maximum area under the curve (AUC) value of the up-regulated model was 0.911, and that of the down-regulated model was 0.896. Additionally, the accuracy values of the down-regulated and upregulated models on independent verification pairs were 0.91 and 0.90, respectively.¹¹⁵ This study is expected to have implications in identifying potential therapeutic targets for the development of antitumor drugs.

Figure 2.

Flowchart of developing models for SM-miRNA regulation prediction. Data set1 was used to construct models to predict the upregulation pairs of small molecules and miRNAs. Similarly, data set2 was used to construct models to predict down-regulation pairs. Reproduced with permission from ref ¹¹⁵. Copyright 2022 Frontiers Media SA.

As ML continues to evolve, its various methods have future directions in various fields. Currently, probabilistic graphical models, neural networks, and other methods based on probability are research hotspots, in addition to the NB algorithm.¹¹⁶ NB models have stable classification efficiency and perform well on small-scale data, handle multiclassification tasks, and are suitable for incremental training. However, it can lead to poor predictions due to the assumed prior model or classification decision errors. Similarly, SVM has developed rapidly in finite dimensions, but further research is needed in infinite dimensions,¹¹⁷ and its robustness¹¹⁸ also needs improvement. RF is capable of efficiently operating on large data sets¹¹⁹ but sometimes results in a large number of decision trees, which enlarges the space and time required for training.¹²⁰ Fortunately, RF will play a significant role in this era of increasing data volume. Later, we will discuss some special ML algorithms that are suitable for small sample applications.

3.2. Artificial Neural Networks

With the continuous development of AI, significant progress has been made in the field of AI. However, in some complex research areas, AI cannot completely replace the human brain in solving complex problems. With increasing exploration by researchers, ANN has proven to be effective in replacing the human brain to solve difficult problems. In 1943, McCulloch et al.¹²¹ proposed the first ANN computational model, called the M-P model, which promoted the development of ANN research. ANN, which is also known as a collection of connected units of artificial neurons, is a framework for many different algorithms from ML. A basic ANN structure usually contains three parts: an input layer, a hidden layer, and an output layer. Due to their advantages of self-learning and powerful computing power, ANNs have been widely used in various fields, such as face recognition,^122–124 medical diagnosis,^125–127 and speech recognition.^128,129 The generalization ability of neural networks is mainly dependent on the size of the training set and the network architecture. Generally, the performance of the neural network improves as the number of samples in the data set increases. In recent years, ANNs have played a vital role in small data set research in the fields of biology and chemistry.¹³⁰ Researchers have applied neural networks to obtain the optimal experimental results. In the following section, we will summarize and analyze the research on ANNs in small data sets.

In the field of drug design, researchers have used ANNs to develop predictors for log P. For instance, Chen et al.¹³¹ developed a predictor for log P using a fully connected ANN model. This study is expected to contribute to the identification of potential therapeutic targets for antitumor drug development. This, in turn, can help reduce time, effort, costs, and attrition rates in drug discovery by enabling the rejection or prioritization of compounds without the need for synthesis and testing. While, in order to explore anticancer properties of thioguanine, Hoseini et al.¹³² applied an ANN approach to generate quantitative structure–property relationships models for log P prediction. Additionally, Dadfar et al.¹³³ developed genetic algorithm–multiple linear regressions (GA-MLR) and genetic algorithm–artificial neural network (GA-ANN) models to predict the log P of sulfonamides. Sulfonamides are compounds with a wide range of biological activities and serve as the basis for several groups of drugs.

In clinical practice, predicting blood-to-plasma concentration ratios is crucial for determining drug administration regimens. However, only a few studies have investigated methods for predicting concentration ratios. In 2021, Mamada et al. developed an concentration ratios prediction model incorporating typical human pharmacokinetics parameters.¹³⁴ They compiled experimental concentration ratio values for 289 compounds, providing reliable predictions by extending the range of application. The authors used human pharmacokinetics parameters, including the volume of distribution, clearance, mean residence time, and plasma protein binding rate calculated from plasma drug concentration and 2702 molecular descriptors to construct a quantitative structure–pharmacokinetics relationship model for concentration ratios. Among the algorithms analyzed, the ANN algorithm had the best performance. After optimizing with six molecular descriptors and log Vd, the correlation coefficient of the model is 0.64, and the root-meansquare error (RMSE) is 0.205, which is better than other concentration ratio prediction methods reported in the past.

In 2022, Mayer et al. investigated the nucleation of dislocations in homogeneous lattices, which is related to small-scale plasticity or ultrafast loading.¹³⁵ They prepared training data using molecular dynamics (MD) simulations and performed polynomial extrapolation beyond the nucleation limit to improve the accuracy of the trained ANN and make theoretical predictions more accurate. The authors considered atomic configurations observed during dislocation nucleation and subsequent development and presented an approximation method that required smaller and simpler MD data for training. Their method gave a strain rate dependence for the nucleation threshold that was close to that of a rigorous theory of dislocation nucleation. A schematic diagram is shown in Figure 3.

Figure 3.

MD assisted ANN prediction of the nucleation of dislocations in homogeneous lattices. (a) Nucleation of a dislocation loop by gradual displacement of a part of the atoms along the loop area. (b) The following mechanical growth of a supercritical dislocation loop by slip of dislocation lines. Reproduced with permission from ref ¹³⁵. Copyright 2022 Elsevier.

The performance of small molecule receptors in organic solar cells is determined by their chemical structure. To avoid trial-and-error based design, multiscale simulation is necessary, which can ultimately save time and resources. In 2022, Mahmood et al.¹³⁶ collected data on 164 small molecule nonfullerene acceptors from the literature. Computational analysis is a quick and efficient way to narrow down potential candidates for synthesis. This work shows that properly regulating sp²-hybrid nitrogen substitution is an effective way to tune the properties of the electron acceptors. This study also demonstrates the potential of multiscale theoretical modeling, which makes it possible to envision structural changes from atomic to molecular levels.

In the future, ANNs will be more frequently applied to neurobiology,¹³⁷ enabling researchers to derive testable insights and predictions from neurobiological experiments. ANNs have already been integrated with other advanced methods, such as fuzzy logic¹³⁸ and wavelet analysis,¹³⁹ to enhance their ability for data interpretation and modeling, as well as to avoid subjectivity in the operation of the training algorithm.¹⁴⁰ ANNs are expected to show their talents in more fields in the future, mainly due to their powerful data processing capabilities. They can outperform almost all other ML algorithms in some cases, such as cancer detection, which is a demanding task,¹⁴¹ where better performance can lead to more people being treated. However, one of the disadvantages of ANNs is data gluttony, as they generally require more data than traditional ML algorithms. Additionally, ANNs have other downsides such as the classic black box problem,¹⁴² as well as being time-consuming and labor-intensive in their training.

3.3. Convolutional Neural Networks

CNN is a valuable tool in the analysis of biological data¹⁴³ and is a type of DL algorithm inspired by the natural visual perception mechanism in biology.¹⁴⁴ LeCun et al. proposed LeNet-5 in 1998 for standard handwritten character recognition.¹⁴⁵ The network structure is relatively complete, which is one of the fundamental components of modern CNNs, making LeNet-5 the beginning of the class of CNNs. Over time, Krizhevsky et al. came up with AlexNet,¹⁴⁶ which performed exceptionally well in image classification. Later, VGG-Net¹⁴⁷ and GoogLeNet¹⁴⁸ were created in the same year and achieved remarkable performance in the ImageNet classification task. It is worth mentioning that ResNet⁹ made a significant innovation in the network structure and pioneering work in computer vision and DL. In addition, DenseNet¹⁴⁹ was proposed, a CNN with dense connections, which further improved the network’s performance. Next, we introduce the CNN structure. CNNs are a collection of neurons organized in interconnected layers, with convolutional, pooling, and fully connected layers.¹⁴³ The convolution layer is used to extract local features, the pooling layer is responsible for significantly reducing parameter size, and the fully connected layer is used to output the desired results, similar to the traditional neural network part.¹⁵⁰ In fact, CNNs have been applied in various fields and have yielded remarkable results, such as image processing,^151–154 action classification,^155–157 NLP,^158–161 physics,^162–164 and more. CNNs are popular in the fields of biology and chemistry for studying quantitative conformational relationships (QSAR). For example, Hu et al.¹⁶⁵ proposed an end-to-end encoder-decoder model and CNN architecture for QSAR prediction, and Karpov et al.¹⁶⁶ constructed a transformer–CNN framework for generating higher quality, interpretable QSAR models. Additionally, Hamza et al.¹⁶⁷ used a CNN model for bioactivity prediction. We next focus on how CNNs excel in predicting biochemical molecules on small data sets.

Drug-induced liver injury poses a significant challenge in drug development and postmarketing safety monitoring, as it can cause clinical trial failures and drug withdrawals. Traditional safety testing methods are inadequate to address this pharmacological problem due to their limited predictive capabilities. In 2020, Nguyen-Vo et al.¹⁶⁸ proposed a novel NLP-inspired computational framework using CNN and molecular fingerprint embedding features to address this issue. The construction of their model is illustrated in Figure 4. Their development set included 1597 samples, consisting of 946 DILI compounds and 651 non-DILI compounds, while their independent test set included 322 samples, including 128 DILI compounds and 194 non-DILI compounds. The study achieved an average accuracy of 0.89, a Matthews correlation coefficient (MCC) of 0.80, and an AUC of 0.96. The results indicate that the proposed model significantly outperformed the latest and best model with a 6.67% improvement in AUC from 0.90 to 0.96. Additionally, the findings suggest that molecular fingerprint embedding features are an effective method for molecular representation in biological research, complementing traditional molecular fingerprinting applications.

Figure 4.

Architecture of a DL model for screening of DILI compounds. The model consists of an embedding layer, a convolutional block and a fully connected block. The fully connected block consists of three fully connected layers. Except for the fully connected blocks in the last layer, the others are designed with batch normalization. Reproduced with permission from ref 168. Copyright 2020 American Chemical Society.

Quantitative structure–activity relationships (QSARs) play important roles in the environmental field. In 2021, Zhong et al.¹⁶⁹ used molecular images combined with CNN to develop QSARs to predict the rate constants of hydroxyl radical generation from compounds. The data set contained 1159 organic compounds, which were initially classified into 357 classes based on all functional groups. However, 250 of the 357 classes contained fewer than three compounds and could not be divided into the training, validation, and test data sets. Therefore, based on functional group similarity, they merged classes with less than 3^–4 compounds with larger groups to form 98 classes.¹⁷⁰ The study developed molecular image-CNN models using transfer learning and data augmentation techniques. These techniques greatly improve the robustness of the model and prediction performance. Experimental results show that the proposed model has a better prediction performance than the model based on molecular fingerprints.

MD simulations are effective in analyzing the transport characteristics of liquids on solid surfaces with different nanometer-scale roughness, but they require high computational costs.¹⁷¹ In 2022, Li et al. proposed a DL encoder-decoder CNN to predict the adsorption density distribution of atoms and organic liquids at various molecular-scale surface roughnesses.¹⁷² The data set consisted of monatomic liquids (sample size: 384) and polyatomic liquids (sample size: 384), with 344 samples in the training set and 40 samples in the test set for both single atoms and multiple atoms. The CNN structure and parameter settings are shown in Figure 5. The proposed method achieved high accuracy in predicting adsorption densities at different microinterfaces with a small data set. The experimental results show that MD and DL methods have good coupling, which can help in designing surface geometry to obtain ideal molecular liquid interface transport characteristics and complement the nanoscale model system for interactive visualization.

Figure 5.

Encoding–decoding CNN construction for the molecular adsorption density prediction. The proposed CNN mainly consisted of four parts: input layer, encoding module, decoding module and output layer. Reproduced with permission from ref 172. Copyright 2022 Elsevier.

In summary, CNN is a great approach for solving small data set problems in various fields, such as pharmacology,^168,173,174 chemical efficacy testing,¹⁷⁵ and protein structure prediction.^143,176 One of the most significant advantages of CNN^177–179 is its ability to identify important features without human supervision. Additionally, CNN is highly accurate at image recognition and classification.¹⁸⁰ Another major advantage of CNN is its weight sharing property, which reduces the amount of computation required compared with regular neural networks. However, CNN also has some drawbacks.¹⁸¹ First, it fails to encode the position and orientation of objects.¹⁸² Second, CNN’s effectiveness depends on having a large amount of training data. Third, CNN tends to be slower due to operations such as maxpool.¹⁸³ Finally, because CNN is made up of multiple layers, the training process can take a long time if the computer lacks a powerful GPU. Despite these drawbacks, CNN has shown promising results when applied to small data sets of biochemical molecules. Nevertheless, researchers need to consider issues such as efficiency, experimental costs, and result generation. After years of research and application, CNN has become one of the representative DL algorithms, reflecting its powerful functions in many aspects. In the future, CNN will have more applications in the field of small molecules and will play a greater role. Thus, further research and exploration are necessary to improve CNN.

3.4. U-Net

Semantic segmentation has been successfully used in various fields, including geological detection, automatic driving,^184,185 and agriculture.¹⁸⁶ It is a fundamental task in computer vision, and its first model, a fully convolutional network, was proposed by¹⁸⁷ in 2015. Since then, several other models have been introduced, such as the U-Net model,¹⁸⁸ SegNet,¹⁸⁹ dilated convolutions,¹⁹⁰ and Deeplab.¹⁹¹ In particular, we focus on U-Net and its application to small data sets in the biochemical molecular field. U-Net belongs to the Encoder–Decoder structure,¹⁹² and its framework is shown in Figure 6c.¹⁹³ The original intention of U-Net was to solve problems in biomedical images.¹⁹⁴ Because of its excellent performance, U-Net has been widely used in the fields of biology and chemistry, such as drug and material design,¹⁹⁵ protein structure prediction,^196,197 as well as other topics such as satellite image segmentation¹⁹⁸ and industrial defect detection.¹⁹⁹

Figure 6.

Submanifold sparse convolution based U-Net in a 2D perspective. The difference between it and traditional 3D-CNN is illustrated in (a) and (b). In (c), we demonstrate the architecture of U-Net. Reproduced with permission from ref 193. Copyright 2022 American Chemical Society.

In the molecular field, the U-Net model has shown promising results for small data sets. In 2021, Nazem et al.²⁰⁰ developed a 3D U-Net model based on voxels for predicting binding sites in protein structures. The algorithm was trained and validated on a subset of scPDB, which is the largest and highest quality binding site database selected from the PDB. To test the model’s performance, the authors used three data sets: Chen11, B210, and DT198. Chen11 contains 251 structures with the maximum number of relevant pockets; B210 is a set of 210 protein structures in the bound state from LIGSITE-csc, and DT198 contains 198 drug target complex structures. The model was also assessed on the B48/U48 database to show its performance on the apo structures of proteins. The evaluation metric used was the F1-score, and the F1-scores for the three data sets were: B210 (0.41), DT198 (0.40), and Chen11 (0.37). All of these scores were higher than those achieved by the LIGSITE-csc and DeepSite methods.

In 2021, Kotowski et al.²⁰¹ proposed a single-sequence-based protein prediction method, called ProteinUnet, which leveraged the U-Net convolutional network architecture. The article aimed to predict protein function and structure from sequence and used protein data sets from CullPDB and named them TR9993 and TS1199. Specifically, TR9993 consists of 9993 different chains from 9622 proteins as the training set, and the test set TS1199 consists of 1199 chains from 1187 different proteins. The authors concluded that their model had better classification accuracy compared to the SPIDER3-single model, and more detailed results are shown in Table 2. This table provides the mean accuracies of Q3 and Q8 predictions at the sequence level in TS1197 and CASP13, along with the standard deviations and p-values of the two-sided Wilcoxon signed-rank test between the models.

Table 2.

Performance in Secondary Structure Prediction by ProteinUnet and SPIDER3-Single on TS1197 and CASP13 According to the Mean Accuracy and SD at the Protein Sequence Level^a

		TS1197			CASP13
		mean %	SD %	p-value	mean %	SD %	p-value
Q3	ProteinUnet	73.53	8.70	0.0152	74.39	8.13	0.0128
	SPIDER3-Single	73.18	9.04		75.12	7.65
Q8	ProteinUnet	61.82	10.86	<0.0001	60.81	12.17	0.8961
	SPIDER3-Single	61.34	11.15		60.81	12.79

^aReproduced with permission from ref 201. Copyright 2021 Wiley.

In 2021, Prasad et al.²⁰² developed an automatic liver parenchyma segmentation network based on the U-Net architecture. The authors used a data set consisting of highly variable venous phase enhanced computed tomography (CT) volumes, with 10 males and 10 females as the source, 75% of whom had liver tumors. However, due to the small size of the data set, the model was overfitting, and the authors had to take some measures, such as reducing the convolution and dropout layers. They also added Gaussian noise to prevent overfitting and solved the problem of inconsistent intensity by pixel normalization. To build a model with better performance, it was important to choose an appropriate loss function. The authors evaluated four loss functions: Dice loss, binary cross-entropy loss, Tversky loss, and focal Tversky loss, and we found that the Dice loss function performed the best, achieving a score of 94.5%. Their work may play a crucial role in assisting oncologists and surgeons with accurate analysis of various pathological conditions, ultimately saving time.

As an improved version of the fully convolutional network model, U-Net has several characteristics that make it suitable for large medical image segmentation.²⁰³ These include multiscale capability,²⁰⁴ simple structure,²⁰⁵ and the use of skip links.²⁰⁶ However, U-Net also has some drawbacks, such as slow running efficiency²⁰⁷ and the limitation of being able to predict on a single scale.²⁰⁸ In the future, supervised,^38,39 semi-supervised,^53,209 and unsupervised learning^48,49 could be potential areas of research for U-Net, as medical image data often lacks sufficient labeled examples. Additionally, the combination of U-Net and AL^210,211 could also be a promising direction for addressing the challenge of data labeling.

3.5. Graph Neural Networks

In recent years, graph neural networks (GNN) have become powerful and practical tools for ML tasks in graph domains. The GNN model was first introduced by Gori et al.²¹² and Scarselli et al.²¹³ and Micheli et al.²¹⁴ developed and improved upon the algorithm. The success of GNN in many domains such as recommender systems,^215,216 computer vision,²¹⁷ and NLP^218,219 is attributed in part to its effectiveness in extracting latent representations from Euclidean data. However, as data are increasingly represented in the form of graphs, including non-Euclidean domains such as e-commerce,^220,221 chemistry,^222,223 and citation networks,^224,225 there is a growing need for GNNs. Additionally, molecular property prediction is a popular application of GNNs, as molecules can be represented as topological graphs, with atoms as nodes and bonds as edges. Currently, the most advanced GNNs can be categorized as GCNs,²²⁶ graph autoencoders,²²⁷ recurrent GNNs, and spatial-temporal GNNs.²²⁸ It is worth noting that there are still open questions about how GNNs handle small data on molecular science or small molecular data that need to be addressed.

In 2021, Yaqing Wang et al.²²⁹ proposed property-aware relation networks which are compatible with existing graph-based molecular encoders to address the limitations of quantitative structure–property relationships and the issue of existing works failing to leverage related graphs among molecules. The overall architecture of property-aware relation networks is shown in Figure 7. The authors conducted experiments on widely used benchmark few-shot molecular property prediction data sets from MoleculeNet:²³⁰ Tox21, SIDER, MUV, and ToxCast, which consist of 8014, 1427, 93127, and 8615 molecules, respectively. The article used the ROC-AUC metric to evaluate the model performance on these benchmark molecular property prediction data sets. Empirical results consistently showed that property-aware relation networks achieved state-of-the-art performance on the few-shot molecular property prediction problem.

Figure 7.

Architecture of a GNN-based classifier for toxicity classification. PAR is optimized over a set of tasks. Within each task ${\mathrm{T}}_{\tau }$, the modules with dotted lines are fine-tuned on support set ${\mathrm{S}}_{\tau }$ and those with solid lines are fixed. A query molecule $x_{\tau ,i}$ will first be represented as $g_{\tau ,i}$ using a graph-based molecular encoder, then transformed to $p_{\tau ,i}$ by our property-aware embedding function. This $p_{\tau ,i}$ further coadapts with embeddings of molecules in ${\mathrm{S}}_{\tau }$ on the relation graph as $h_{\tau ,i}$, which is taken as the final molecular embedding and used for class prediction. Reproduced with permission from ref 229. Copyright 2021 NeurlPs.

In 2020, Pappu and colleagues²³¹ investigated the use of pretraining and the meta-learning technique MAML (as well as variants FO-MAML and ANIL) to enhance the performance of GNNs via transfer learning from related tasks, allowing for their use even in settings with limited data availability. The authors created a new data set comprising 645 binary classification tasks from the ChEMBL database, filtered for five distinct task types. The study found that the performance of the GNN model was initially lower than fingerprint methods but significantly improved with the use of MAML and FO-MAML, outperforming both fingerprint and pretraining methods as measured by the area under the precision-recall curve of the models. The results suggested that meta-learning can improve the use of GNNs in low-data settings compared to fingerprint methods.

Generally, existing DL methods for molecular property prediction require large training data sets for each property, which limits their performance in cases where there is only a limited amount of experimental data, especially for new molecular properties. To address this issue, Zhichun Guo et al. proposed Meta-MGNN, a novel model for few-shot molecular property prediction in 2021.²³² Meta-MGNN’s skeleton framework can be seen in Figure 8. To evaluate the performance of Meta-MGNN, the authors used the Tox21 and Sider data sets, which consist of 7831 and 1427 samples, respectively. The overall performance of all methods was evaluated using the AUC metric, and the results showed that Meta-MGNN outperformed all baseline models on both the Tox21 and Sider data sets. Specifically, for 1-shot learning, the average improvements were +1.04% and +1.80% on the Tox21 and Sider data sets, respectively, and +0.84% and +1.87% for 5-shot learning.

Figure 8.

Overall framework of a Meta-MGNN model for toxicity predictions: It first samples a batch of training tasks. For each task, there are a few data examples in the support set. These examples are fed into a GNN parametrized by $\theta$. Then the support loss ${ℒ}_{\text{support}}$ is calculated and utilized to update the GNN parameters to ${\theta }^{\prime }$. Next, the examples in the corresponding query set are fed into the GNN parametrized by ${\theta }^{\prime }$ and calculate the loss ${ℒ}_{\text{query}}^{\prime }$ for this task. The same process is repeated for other training tasks. Later, we compute the summation of ${ℒ}_{\text{query}}^{\prime }$ over all sampled tasks and use it to further update the GNN parameters for testing. Reproduced with permission from ref 232. Copyright 2021 Web of Conferences.

GNNs have become widely used not only in the fields of biology and chemistry, such as protein–protein interaction networks,²³³ protein structure prediction,²³⁴ and chemical property estimation²³⁵ but also in various other ML applications such as reinforcement learning,^236,237 semi-supervised,^238,239 and unsupervised^240,241 learning. However, due to the complexity of the graph structures, GNN models are not always effective in all graph conditions. To address this issue, several future research directions have been proposed. For instance, the robustness of GNN models should be enhanced because they are vulnerable to adversarial attacks.²⁴² Moreover, because GNN models are often treated as black-boxes, there is a need for improved interpretability on graphs. Thus, research in generating example-level explanations for GNN models has been proposed.^243,244 Additionally, graph pretraining²⁴⁵ and the challenges associated with complex graph structures are also important research directions. Nonetheless, GNNs also have certain limitations such as their performance being limited by their depth and width,²⁴⁶ their inability to work with insufficient data,²⁴⁷ and issues related to high computational costs.

3.6. Long Short-Term Memory

LSTM is a type of RNN that addresses the vanishing gradient problem during training and is capable of learning long-term dependencies. It was introduced by Hochreiter et al.²⁴⁸ in 1997 and has been refined and applied in various fields. Unlike other DL models, the LSTM is specifically designed to handle long-term information without incurring significant cost. LSTM employs back-propagation as its main parameter training algorithm, which involves four steps: forward pass to calculate the output values, error computation using a loss function, backward error propagation among the neurons, and weight parameter updates. LSTM has been successful in a variety of biological and chemical fields, including chemical–protein relation extraction,²⁴⁹ chemical substance classification,²⁵⁰ and drug molecular design.²⁵¹ Additionally, it has been used in fields like imaging,^252,253 speech recognition,^254,255 NLP,²⁵⁶ and more. However, when the data set is small or has few labeled samples, using LSTM may not always yield desired results. This is particularly relevant in bioinformatics, biochemistry,²⁵⁷ and other fields where data sets typically have fewer than 5000 elements.

In the field of protein research, predicting the structure of proteins is essential for understanding their function and designing drugs. Traditional techniques for protein structure prediction are often time-consuming and expensive, and developing new advanced methods remains a major challenge. The secondary structure of proteins is critical for analyzing protein function and designing drugs. Various computational methods have been proposed to improve the performance of protein secondary structure prediction. In 2019, Guo et al.²⁵⁸ proposed a novel deep neural network approach called deep asymmetric convolutional LSTM neural network (DeepACLSTM) for predicting protein secondary structure from protein sequence features and profile features. DeepACLSTM utilized the eigenvector dimension of the protein feature matrix to effectively combine asymmetric CNNs²⁵⁹ and bidirectional LSTM (BLSTM) neural networks to predict protein secondary structure. It comprised three main modules. In this paper, DeepACLSTM was compared with several methods such as SSpro8,²⁶⁰ conditional neural field (CNF), DeepCNF (CNF based on DL),²⁶¹ and CBRNN.²⁶² To evaluate the performance of DeepACLSTM, experiments were conducted on three publicly available data sets: CB513, CASP10, and CASP11. The results demonstrated that DeepACLSTM outperformed the state-of-the-art baseline on all three data sets.

In the field of medicinal science, cancer remains a significant threat to human health. Anticancer peptides (ACPs) present a promising avenue for cancer treatment and offer many advantages. However, traditional experimental methods for identifying novel anticancer peptides can be costly and inefficient. In 2019, Yi et al.²⁶³ proposed a DL-LSTM neural network model, ACP-DL, for effectively identifying new anticancer peptides. The authors combined binary contour features and a k-mer sparse matrix of simplified amino acid letter features to construct an efficient feature representation that maximized the use of peptide sequence information. Additionally, a deep LSTM model was utilized to automatically learn to discriminate between anticancer and ordinary peptides. The workflow of this approach is shown in Figure 9. To evaluate the performance of the method, the authors used the ACP740 (with a sample size of 740) and ACP240 (with a sample size of 240) data sets and compared the results using 5-fold cross-validation, which verified the state-of-the-art performance of ACP-DL.

Figure 9.

Illustration of a LSTM architecture utilizing k-mer sparse matrices and binary contour features for predicting anticancer peptides.²⁶³

In 2017, Li et al.²⁶⁴ proposed a predictive model called ProDec-BLSTM for investigating protein remote homology detection. The model included an input layer, a BLSTM layer, a time-distributed dense layer, and an output layer. The framework diagram for ProDec-BLSTM is shown in Figure. 10. The performance of the model was evaluated using the SCOP data set, which had a sample size of 4019. ProDec-BLSTM was compared with GPkernel,²⁶⁵ GPextended,²⁶⁵ GPboost,²⁶⁵ SVM-Pairwise,²⁶⁶ Mismatch, eMOTIF,²⁶⁷ LA-kernel,²⁶⁸ PSI-BLAST,²⁶⁹ and LSTM²⁷⁰ on the same data set. ProDec-BLSTM achieved a mean receiver operating characteristic curve (ROC) of 0.969 on the evaluation metric, which was higher than those of the other methods.

Figure 10.

Structure of ProDec-BLSTM for protein remote homology detection. The input layer converts the pseudo proteins into feature vectors by one-hot encoding. Next, the subsequences within the sliding window are fed into the BLSTM layer for the extraction of the sequence patterns. Then, the time-distributed dense layer weighs the extracted patterns. Finally, the extracted feature vectors are fed into an output layer for prediction. Reproduced with permission from ref 264. Copyright 2017 Springer Nature.

In addition to the previously mentioned applications of LSTM, successful applications have been documented in the fields of biophysics and bioinformatics. Various variant models have also been proposed by combining LSTM techniques to improve accuracy, such as the flow-based LSTM model proposed by Gers et al.²⁷¹ in 2000. Recently, Zhu et al.²⁷² proposed a variant model called ACP-check, which utilized BLSTM networks and multifeature fusion. The model extracted time-dependent information features from peptide sequences by using a BLSTM network and combined them with amino acid sequence features. To validate the performance of the model, six benchmark data sets were selected, including ACPred-Fuse, ACPred-FL, ACP240, ACP740, main, and alternate data sets of AntiCP2.0. ACP-check achieved prediction accuracies of 0.91, 0.91, 0.90, 0.87, 0.78, and 0.93, respectively, with improvements ranging from 1% to 49%. These results demonstrated the excellent predictive performance of ACP-check. Other improved models, such as Bidirectional LSTM,²⁷³ have also been proposed for short-term load forecasting. These successful examples of LSTM-based variant models indicate that the use of LSTM techniques is not only effective in improving experimental results but also has a wide range of applications. However, further research is needed to determine the optimal method for combining LSTM with small data sets.

3.7. Generative Adversarial Networks

Although DL has made significant breakthroughs in various research fields, the quality and quantity of data often affect its results. In 2014, Goodfellow et al. proposed an innovative GAN model.²⁷⁴ Unlike other DL algorithms, GAN has a discriminant model composed of two main parts: the generator and the discriminator. The generator is responsible for creating synthetic data samples, while the discriminator tries to differentiate between real and synthetic samples. These two parts compete against each other during the training phase, where the generative model learns the distribution of the sample data. Note that the discriminative model is often a dichotomous classifier used to distinguish between real and generated data. The flowchart of the GAN framework is shown in Figure 11.

Figure 11.

Framework of a general-purpose GAN model. It consists of a discriminator and a generator.

With the rapid development of GAN, many generalizations have been proposed to improve the original method. Examples of these include AdaGAN,²⁷⁵ MADGAN,²⁷⁶ PacGAN,²⁷⁷ D2GAN,²⁷⁸ SGAN,²⁷⁹ and DCGAN.²⁸⁰ GAN and its extensions are often used to augment data sets and mitigate overfitting issues in downstream ML and DL tasks. In the following sections, we summarize how GANs have been utilized in small data sets of chemical and biological molecules.

In 2019, Han et al.²⁸¹ proposed protein log S generative adversarial nets (ProGAN), a data augmentation method to address the issue of insufficient data for protein log S prediction. The data set comprised 3148 samples from the ESOL database,²⁸² and the evaluation metric used was the coefficient of determination, R². ProGAN was employed solely for data augmentation and combined with the DNN method to enhance the prediction performance. The optimal results were achieved using the sigmoid activation function.²⁸³ The test set R² for DNN was 0.40 ± 0.0074, and for DNN+ProGAN, it was 0.42 ± 0.0067. The DNN+ProGAN method with the sigmoid activation function yielded the highest experimental result, with an R² of 0.45 ± 0.0018. The results of this work have the potential to enhance the production yield of recombinant proteins in biocatalysis applications.

In 2019, Liu et al.²⁸⁴ developed a model that combined GAN and deep neural networks (DNN) for multiple classifications with small cancer-staging sample sizes. First, the original data were split into a training set and a test set, and the GAN was trained using the training set to generate synthetic samples that expanded the training set. Then, the DNN classifier was trained using the synthetic samples, and the classifier was tested with the test set using different metrics to verify the effectiveness of the method. The data set used in the experiment had less than 100 samples, which were divided into a training set (60%) and a test set (40%). Classical ML methods such as RF²⁸⁵ and NB were used as a comparison, and the SMOTE²⁸⁶ method was used to generate oversampled samples to train the classifiers. In the WGAN-based framework, a large number of synthetic samples generated by WGAN were used to train the classifiers, and then the classifiers were validated with real samples. The experimental results were presented in Table 3. The evaluation metrics used were accuracy, F-measure (the harmonic mean of precision and recall),²⁸⁷ and the geometric mean of recall,²⁸⁸ which demonstrated that the proposed method substantially improved the results of the classification experiments under the condition of increasing the number of synthetic samples.

Table 3.

Performance of Different Classification Strategies for Cancer-Staging Data²⁸⁴

	RF	NB	RF with SMOTE	NB with SMOTE	DNN with SMOTE	RF with WGAN	NB with WGAN	DNN with WGAN
accuracy	0.5484	0.3226	0.5806	0.5484	0.6452	0.2903	0.6129	0.7097
F-measure	0.5673	0.1220	0.6254	0.5600	0.6605	0.2931	0.6260	0.7007
g-mean	0.4550	0	0.4811	0.5233	0.6332	0.2627	0.6043	0.6839

In the field of cancer research, the issue of insufficient data often leads to poor performance of ML models. To address this problem, Wei et al. proposed Gene-GAN in 2022,⁷⁴ a model for classifying cancer data. As the data set contained less than 500 samples, they used GAN to augment the data and employed the reconstruction loss to stabilize model training, resulting in high-quality generated samples. The excellent performance of Gene-GAN was demonstrated by comparing it with different classifiers in Table 4, which also highlighted the importance of data augmentation using GAN. In the table, Gene-GAN (mixed) indicated that the generated data was used in combination with the original data, while Gene-GAN (nonamplified) meant that the augmented data was not used. The experimental results confirm that the generative model is an effective solution to the problem of insufficient sample size.

Table 4.

Classification Results for Different Classifiers for Cancer Data^a

classifiers	accuracy (mean)
Decision tree	0.608
KNN (k = 3)	0.864
SVM	0.84
VGG	0.781
ResNet	0.849
Gene-GAN (nonamplified)	0.85
Gene-GAN (mixed)	0.892

^aReproduced with permission from ref 74. Copyright 2022 Springer Nature.

In 2020, Hsu et al.²⁸⁹ proposed the Wasserstein-based data augmentation algorithm, which utilized GANs to augment data during model training. The authors conducted experiments on a breast cancer data set containing 582 samples. The results of Wasserstein-based data augmentation demonstrated higher accuracy, AUC, and concordance index values compared to those of the data augmentation algorithm. Specifically, the accuracy value was 0.6726 ± 0.0278, the AUC value was 0.7538 ± 0.0328, and the concordance index value was 0.6507 ± 0.0248. The results suggest that GANs can be effectively used to train deep models in medical applications, even when limited data is available.

In 2021, Li et al.²⁹⁰ presented a derivative model, BrainNetGAN, based on GAN for the synthesis of conditional brain networks. The brain network matrix was used as input to generate a fake brain network connection matrix through BrainNetGAN, and then, the potential distribution and topological characteristics of real brain network data were inferred. The experiments evaluated the data augmentation performance of BrainNetGAN and compared its results with the experimental results of Baseline without augmented data. Specifically, BrainNetGAN attained an accuracy of 0.812, which was higher than the baseline of 0.791.

The experiment conducted by Lin et al.²⁹¹ in 2021 aimed to develop a sequence-based binary classifier to determine whether short peptides exhibited antiviral activity. The antiviral data set used in the experiment consisted of 2934 samples. To address the issue of imbalanced data, the authors employed a GAN model to augment the number of positive data samples, which were then added to the original data set. As a result, the model achieved an accuracy of 84% in the final prediction, which outperformed the accuracy achieved using the original data set without augmenting the data generated by the GAN method.

Nowadays, GAN has gained popularity in both academia and industry due to its numerous applications, not only in the fields of peptide and protein design,²⁹² chemical material design,^293,294 and medicine,^295–297 but also in image genersion, among others. These studies demonstrate the broad range of applications of the GAN methods. Despite their significant success, GANs still have shortcomings in various research fields. For example, the interpretability and controllability of GANs have not been fully understood, and further research on these aspects will remain crucial in the future. Additionally, GANs often suffer from poor stability, which can lead to model collapse.³⁰⁷ Therefore, future research on how to prevent model collapse during GANs training will be important.

3.8. Autoencoders

In recent years, NLP models have become increasingly popular. Among them, (variational) autoencoder (VAE) is considered to be one of the most promising techniques for unsupervised learning. VAE was proposed by Kingma et al.³⁰⁸ in 2013. It not only plays an important role in generating data but also has a wide range of applications in imaging and other fields. With the continuous development of VAE, its structure has become more flexible, and derivative models based on variational autoencoder-based models have emerged. For example, the conditional variational autoencoder was proposed by Makhzani et al.³⁰⁹ in 2015. In 2017, Bao et al.³¹⁰ proposed a model conditional variational autoencoder-GAN combining VAE and GAN for synthesizing images in fine-grained categories, such as faces of a specific person or objects in a category. Other examples include a variational loss autoencoder,³¹¹ multistage variational autoencoder,³¹² and Wasserstein autoencoder.³¹³

In the field of molecular generation, various generative models based on variational autoencoders have been proposed, such as Graph Flow-Variational Autoencoder (GF-VAE),³¹⁴ which combines VAE and the normalized model to generate molecular maps at once. Through the use of variant models of the variational autoencoder, it is evident that VAE has been extensively applied in many research fields. While traditional autoencoders describe the difference of the latent space using numerical methods, VAE models the difference of the latent space using probabilistic distributions. It models the relationship between latent variables and input data from a probabilistic perspective to complete the task of data generation and solve the problem with very few training samples. The model structure of VAE is mainly composed of two parts: the inference network (i.e., encoder) and the generation network (i.e., decoder). The basic process is to map the samples to the latent variables of the low-dimensional space through the encoding process and then restore the hidden variables to the reconstructed samples through the decoding process. The following section summarizes the applications of (variational) autoencoders to small data sets in scientific research.

In 2019, Ohno et al.⁶⁴ used variable self-sorting encoders as a generative model for data augmentation to address the problem of small data volumes in regression tasks. The study utilized seven small data sets of regression type. First, the original data were divided into training and test data, and the generated model was trained on the training data. Next, sampling was carried out using generative models based on ratios. The generated samples were then trained on the regression model along with the original training data. Finally, the RMSE of the test data on the regression model was calculated. Several models were set up for comparison, including kernel density estimation using Gaussian kernel function (KDE), Variational autoencoder (single task learning (VAE)), VAE with linear regression (multitask learning), VAE with nonlinear regression (multitask learning), and denoising autoencoder with MCMC. In evaluating the test data, samples generated by the model were used, with the sample size to training data size ratio ranging from 0 to 1. Changes in the RMSE were evaluated according to the increase in the size of the training data. The experimental results for the ION data set were used as an example in the paper. The RMSE values of the KDE, VAE with linear regression, VAE with linear regression, VAE with nonlinear regression, and DAE-A models were 0.83080, 0.86738, 0.86181, 0.86258, and 1.07335, respectively. The RMSE values for the four models improved with the increasing ratio, except for DAE-A, which may be because the generated samples were highly similar to those in the training data.

In chemistry and biophysics, an issue frequently encountered is the imbalance between the number of available training and test samples. In 2022, Wei et al.⁴¹ proposed a solution that combined a variable self-division encoder and GAN algorithm for data augmentation to address this problem. The authors demonstrated that the R² values of several models including ANN, VAE+ANN, GAN+ANN, RF, VAE+RF, and GAN+RF were 0.57, 0.71, 0.59, 0.89, 0.94, and 0.59, respectively. These results suggest that the proposed approach can improve the performance of the task by enhancing the balance between training and test data sets.

In 2021, Feng et al.³¹⁵ developed a network analysis of cocaine dependence targets that involved more than 450 proteins, including dopamine (DAT), serotonin (SERT), and norepinephrine (NET) transporters. However, the available ligand binding data sets for many of these targets were limited. To improve the accuracy of their ML/DL models, the authors constructed autoencoder-assisted multitask ANN models, as depicted in Figure 12. This method was employed to facilitate drug repositioning and side effect analysis.

Figure 12.

Workflow of an autoencoder-assisted multitask ANN model for enhancing small data sets inferred by interactomics networks of cocaine addition targets.³¹⁵ (a) Sequence-to-sequence autoencoder model is used to create uniform features for different data sets. BLSTM and LSTM are used in encoder and decoder networks, respectively. (b) An MT-DNN model is connected to the autoencoder for regression predictions.

Nowadays, variational autoencoders are widely used not only in the field of generative models for sample design of chemical molecules^316–318 but also in other areas such as imaging³¹⁹ and text generation.^320,321 Variational self-encoders are commonly used in image and biomolecular research to generate new molecular samples. However, there are still some issues with VAEs, such as the generation of noisy data. Additionally, most VAE structures struggle with generating high-resolution image samples, making them less effective in this area compared with GAN-based generative models. As a result, VAEs are often used as feature extractors in image and molecular science.³¹⁵ However, in NLP, VAE-like models are capable of generating more coherent language samples than GANs and require only simple structures to produce fluent language, highlighting the advantages of VAEs in this field.

3.9. Transformers

SSL³²² is a type of unsupervised learning that extracts supervised signals from unlabeled data, which can be used to learn intrinsic constitutional rules and obtain desirable representations using neural networks. Because the supervised information in SSL is not manually annotated, it can be considered a branch of unsupervised learning. SSL is often the first choice for researchers to avoid the high cost of data annotation and the poor performance of traditional unsupervised learning. SSL was initially applied in computer vision and NLP^323,324 that requires large data sets for accurate representation learning. As SSL advances, it has been utilized to predict molecular properties.^325,326 For example, it can extract features from unlabeled molecular data.^28,327 Likewise, SSL can also extract features from genome data³²⁸ to predict genome function. Recently, many studies have shown that SSL can alleviate the problem of few samples or insufficient supervised information, making it widely applicable in image classification,^329,330 recommender systems,³³¹ protein analysis and design,³³² speech recognition,³³³ and other fields. For small data sets of biochemical molecules, many researchers have proposed SSL methods to process them.³³⁴ In the following discussion, we will focus on the applications of SSL to small data sets of biochemical molecules.

In recent years, SSL methods have gained popularity in drug discovery.^336,337 In 2020, Shen et al.³³⁵ proposed an SSL method called Motif Learning GNN (MoLGNN), which was trained on unlabeled chemical data to improve drug screening performance. The method was tested on three data sets, JAK1, JAK2, and JAK3, and compared to the results of three other methods in Table 5. “Non-MoLGNN” referred to a network trained using standard supervised classification methods without pretraining, ”GINVAE” indicated a procedure that pretrained the network using GINVAE and then fine-tuned it, and ”Motif Only” meant a procedure that pretrained and then fine-tuned the network using a Motif learning network. From the table, it was found that the MoLGNN method produced superior results compared with the other methods. The results also suggested that MoLGNN can be applied to a range of machine learning tasks in chemistry, even in scenarios where high-quality labeled data was limited.

Table 5.

Comparison of the Results of Three Kinase Data Sets on Different Methods^a

	JAK1		JAK2		JAK3
data set	ROC (%)	AP (%)	ROC (%)	AP (%)	ROC (%)	AP (%)
Non-MoLGNN	91.8 ± 0.6	85.4 ± 0.7	87.2 ± 0.3	85.5 ± 0.8	81.2 ± 2.5	80.2 ± 2.0
GINVAE Only	94.2 ± 0.1	88.6 ± 0.2	88.0 ± 0.9	87.3 ± 0.4	84.6 ± 0.2	84.7 ± 0.8
Motif Only	93.4 ± 0.4	87.8 ± 1.1	85.8 ± 0.9	87.5 ± 0.2	86.1 ± 0.1	86.7 ± 0.4
MoLGNN	94.5 ± 0.3	89.2 ± 0.5	89.6 ± 0.2	89.9 ± 0.4	89.2 ± 0.2	89.5 ± 0.2

^aReproduced with permission from ref 335. Copyright 2020 NeurIPS.

In 2021, Chen et al.³³⁴ proposed an algebraic graph-assisted bidirectional transformer model for predicting molecular properties. The model was composed of four modules: an AG-FP generator (represented by the blue rectangle), a BT-FP generator (represented by the orange rectangle), a feature combination module using RF (represented by the green rectangle), and a downstream ML module (represented by the pink rectangle), as shown in Figure 13. The creation of BT-FPs involved two steps: training based on SSL (with a large amount of unlabeled input data) and task-specific fine-tuning. The RF algorithm was utilized to compute and rank the importance of the combined features, providing optimal features for the downstream ML algorithms. Experimental results demonstrated that the model proposed obtained the best predictions on the data sets LD50, LC50 and FDA, compared to existing advanced models, like ESTDS,³³⁸ MACCS,³³⁹ FP2,³³⁹ HybridModel,³⁴⁰ BTAMDL2,³⁴¹ ESTD-1,³⁴² Daylight-MTDNN,³³⁹ XLOGP3,³⁴³ and Estate2,³³⁹ as indicated by the squared value of the Pearson correlation coefficient (R²).

Figure 13.

Illustration of the AGBT model. For a given molecular structure and its SMILES strings, AG-FPs are generated from an element-specific algebraic subgraphs module and BT-FPs are generated from a deep bidirectional transformer module, as shown inside the dashed rectangle, which contains the pretraining and fine-tuning processes, and then finally completes the feature extraction using task-specific SMILES as input. Then the RF algorithm is used to fuse, rank, and select optimal fingerprints (AGBT-FPs) for ML.³³⁴

In 2022, Yang et al.³⁴⁴ proposed a multitask SSL framework called SSLDR to tackle the label sparsity problem in computational drug repositioning and accelerate the drug development process. The experiments were conducted on three real-world data sets, namely, Gottlieb, Cdata set, and DNdatase.³⁴⁵ The prediction results demonstrate that SSLDR not only enhances the generalization performance of the ”drug-disease association prediction” task but also leverages a multi-input decoder to improve the autoencoder’s capability to discover potential factors of drugs or diseases. Additionally, the results reveal that the SSLDR model outperforms other methods on all three data sets.

SSL has gained popularity not only in the prediction of chemical molecule properties, as evidenced by several studies,^346–348 but also in other fields such as protein^349,350 and drug design.³⁵¹ Uncovering valuable information from unlabeled data has been a vital research area, and SSL has played a critical role in this endeavor. The most significant advantage of SSL is its ability to achieve good performance without a vast number of labeled samples, which reduces labeling costs and saves time. However, SSL often requires significant memory resources during training and demands high hardware requirements. Moreover, SSL is faced with numerous challenges, such as extracting intrinsic representations from large quantities of unlabeled data and evaluating the accuracy of such representations, which are essential directions for future SSL research.

3.10. Reinforcement Learning

ML has become a ubiquitous computational method in research, and reinforcement learning (RL)^352,353 plays a significant role in it. RL studies the way natural and artificial systems can learn to predict the consequences of and optimize their behavior in environments where actions lead them from one state or situation to the next and can lead to rewards and punishments. A common model for RL is the standard Markov Decision Process.^354,355 RL can be divided into model-based RL³⁵⁶ and model-free RL,^357,358 as well as active RL³⁵⁹ and passive RL.³⁶⁰ DL models can also be used in RL to form deep RL (DRL).^361,362 These methods have been applied to diverse fields in biology and chemistry, including drug discovery,^363,364 protein design,^365,366 and chemical engineering.^367,368 They have also been used in image recognition^369,370 and financial markets.³⁷¹ Next, we will summarize how RL can be applied to the study of biochemical molecules in the context of small data sets.

In the field of drug discovery, evaluating compounds from libraries is one of the most time-consuming tasks. In 2022, Dou et al.³⁷² proposed a ML model suitable for small data sets to predict the inhibition constant (K_i) and half-maximal inhibitory concentration (IC₅₀) of compounds. The prediction task was first transformed into a simple binary classification task, and then the training data set was expanded as the original sample size was small. The paper also employed the reinforcement learning method for feature selection, as illustrated in Figure 14. Lastly, the authors used a particle swarm optimized SVM for the binary classification task, denoted as SVM+. The sample size of the K_i-related data set was 44, and that of the IC₅₀-related data set was 36. Among the classification results, the accuracy of SVM+ on the K_i data set was 0.8074, while the accuracy of traditional SVM, Gaussian NB (GNB), KNN, and RF were 0.7942, 0.7150, 0.7467, and 0.7309, respectively. Moreover, the accuracy of SVM+ on the IC₅₀ data set was 0.8262, while the accuracy of traditional SVM, GNB, KNN, and RF were 0.7943, 0.7411, 0.7731, and 0.7304, respectively. Based on the experimental results, the proposed model outperformed other comparison methods.

Figure 14.

RL model for feature selection in protein–ligand binding. The DQN based reinforcement learning is used to further select features to train a classifier. We formulate a new reward function to balance classification accuracy and number of features. The action set contains two basic operations, adding and deleting based on the χ² test, to search for the optimal state. Reproduced with permission from ref 372. Copyright 2022 Elsevier.

In the field of RNA research, it is important to determine the relationship between MicroRNA and diseases to improve the treatment of complex diseases. In 2021, Cui et al.³⁷³ presented the RFLMDA model by combining the Q-learning algorithm³⁷⁴ and RL. The RFLMDA model fused three submodels, namely CMF,³⁷⁵ NRLMF,³⁷⁶ and LapRLS,³⁷⁷ together by the Q-learning algorithm to obtain the optimal weights S. The data sets used in the experiments included MicroRNAs (with 495 samples), diseases (with 383 samples), and MiRNA–disease associations (with 5430 samples). The performance of RFLMDA was evaluated using 5-fold cross-validation and local validation in the experiments. Finally, the RFLMDA model was compared with other methods using the evaluation indexes AUC and AUPR. The experimental results showed that the AUC value of RFLMDA reached 0.9416, while the AUC values of CMF, NRLMF, and LapRLS were 0.9091, 0.9315, and 0.9367, respectively. These results demonstrate that the RL-based approach can achieve good performance on small data sets.

In 2021, Pereira et al.³⁷⁸ introduced a new approach to optimize the generation of compounds that considered their biological properties and bioavailability through a DRL framework. The framework, illustrated in Figure 15, integrated several technologies, such as DL, multiobjective selection, and RL, with RL being the cornerstone. The RL algorithm updated the properties of the generated molecules by maximizing the reward function. A blood–brain barrier predictor was trained with a data set of 4534 molecules collected from various sources, and canonicalized SMILES were used to represent the molecules. Two descriptors were combined with two different oversampling methods to evaluate the performance of the model’s performance. The accuracy of SMILES+ADASYN and SMILES+SMOTE was 0.924 and 0.913, respectively, while the accuracy of extended-connectivity fingerprint (ECFP)+ADA-SYN and ECFP+SMOTE was 0.935 and 0.944, respectively. The experiments demonstrated that this approach can achieve excellent performance, even with a small number of samples.

Figure 15.

General framework of molecular generation contains 4 DL modules: an unbiased generator (A), a biased generator (B) which shares the same architecture, and two QSAR models for predicting the binding affinity (C) and BBB permeation (D). The DL modules were interconnected by a policy-based reinforcement Learning approach (G) applied with a particular exploration/exploitation strategy (F) based on a multiobjective reward function (E). Reproduced with permission from ref 378. Copyright 2021 Oxford University Press.

In addition, in the field of cancer research, molecular-based cancer classification has become a hot research topic. In 2022, Prathik et al.³⁷⁹ proposed a DRL model for efficient analysis of gene expression data to identify cancer types. The DRL model can easily predict cancer types from gene data sets, even with multiple classification labels. Each class was identified by the deep neural network and continuous estimation using the Q-learning method in RL. Three gene expression data sets were used in this study: glioblastoma data set (with 50 samples), brain tumor data set (with 40 samples), and lung cancer data set (with 34 samples). The principal component analysis algorithm³⁸⁰ was used to analyze the data sets and extract the features, and then the DRL model was used for classification experiments. The DRL model was also compared with other classifiers such as ANN, RF, and SVM. The accuracy of the DRL model in the breast cancer, glioblastoma, and Llung cancer data sets was 98.3%, 99.2%, and 97.34%, respectively, which outperformed the other classifiers. The experimental results are summarized in Table 6, indicating that the DRL model is a useful tool in cancer classification tasks.

Table 6.

Comparison of Prediction Accuracy on Different Data Sets^a

data sets algorithms	breast cancer (%)	glioblastoma (%)	lung cancer (%)
proposed DRL model	98.3	99.2	97.34
SVM	91.32	92.34	93.42
RF	78.9	81.23	82.34
ANN	94.5	93.47	94.5

^aReproduced with permission from ref 379. Copyright 2022 Springer Nature.

RL is a versatile method with applications in various fields, including ethics,³⁸¹ drug design,^382,383 psychology,^384,385 and control theory.^386,387 RL has garnered interest from researchers due to its ability to solve complex scenarios that cannot be tackled by traditional methods, as many problems can be converted to a Markov decision process and solved by using RL. However, RL also has some drawbacks. First, the learning efficiency of RL can be low, as seen in algorithms such as OpenAI Five³⁸⁸ and AlphaZero.³⁸⁹ These issues can be addressed using transfer learning³⁹⁰ or replay buffers (also known as experiential replay).³⁹¹ Second, RL often requires high-quality data and involves a large number of computational processes. Lastly, RL’s greatest feature is its generality, with a generic algorithm capable of learning almost anything. Despite the challenges associated with RL, continued research and development in this field will ensure its widespread use in various research domains.

3.11. Transfer Learning

One possible solution for scarce training data is transfer learning. This technique can address the problem of difficult label acquisition. The term ”transfer learning” was formally introduced by the U.S. Department of Defense Advanced Research Projects Agency in 2005 and has been used earlier under different names in various research areas. Yang et al.³⁹² later provided a detailed introduction to the development, definition, classification, and application of transfer learning. Overall, transfer learning is the application of knowledge, patterns, or distributions learned on one task to different but related tasks,³⁹³ which includes two important concepts: domain and task. The domain could be seen as a particular field at a given moment in time, and the task is to determine what needs to be done. Transfer learning is usually suitable for situations where the source domain has a relatively large amount of data, and the target domain has a small amount of data. The stronger the correlation between the source and target domains, the better the predictive performance will be obtained. This technique reduces the need and effort to (re)collect a large training set, thus mitigating the limitation of small data sizes. In 2019, Jang et al.³⁹⁴ addressed the question of what content to migrate and where to migrate it for transfer learning. As the application of transfer learning continues to expand, it is being used in many areas such as computer vision,^395–397 human–computer interaction,³⁹⁸ text classification,^399–401 target recognition,^402,403 protein analysis,^404,405 and others. Transfer learning has proven to be a powerful technique in biology and chemistry, with applications in gene expression data analysis^406,407 and neuroscience research.⁴⁰⁸ Researchers use their data sets to train transfer learning models and study their structure and function. In addition, drug molecular data can be used to predict the properties and activities of drug molecules, which can lead to the discovery of new drugs.^409,410 Fields that suffer from insufficient data or inadequate data annotation include research areas for rare diseases such as acute promyelocytic leukemia and acromegaly.⁴¹¹ Below, we summarize how transfer learning can be applied to predict biochemical molecules with small data sets.

In 2019, Ye et al.⁴¹² proposed a method that combines transfer learning and multitask learning (DeepPharm) to enhance the generalization ability of the model for scarce training data sets. The authors utilized four different data sets with small sample sizes: oral bioavailability (sample size: 410), plasma protein binding rate (sample size: 769), apparent volume of distribution at steady-state (VDss) (sample size: 412), and elimination half-life (sample size: 969). The predictive performance of DeepPharm was compared with other methods, such as SVM. The SVM method achieved an accuracy of 23% and a mean absolute error (MAE) value of 0.34 for the bioavailability data set. In contrast, DeepPharm increased the accuracy to 28% and decreased the MAE value to 0.31, which suggests that DeepPharm can be further employed in drug discovery and development.

In 2020, Sharifi-Noghabi et al.⁴¹³ introduced an adversarial inductive transfer learning technique, which combined adversarial training with inductive transfer learning, for solving problems in pharmacogenomics applications that required adaptation in both input and output spaces. The GDSC and GSE28796 data sets used in this study had small sample sizes of 829 and 12, respectively. Impressively, the adversarial inductive transfer learning technique improved the AUROC value up to 51% and 45% compared to the ProtoNet⁴¹⁴ and ADDA method,⁴¹⁵ respectively.

In addition, Bai et al.⁴¹⁶ developed a sequence-to-sequence (seq2seq) transfer learning method that introduced transfer learning into reverse synthesis analysis, as illustrated in Figure 16. The method utilized an unclassified large data set, USPTO 380K, for pretraining the model, followed by continuous training and reverse synthesis testing on the small data set USPTO-50K. Then, the transfer learning was combined with the seq2seq or transformer model for validation. The accuracy value obtained using the seq2seq-transfer learning method was 72.1%, which was higher than 65.9% obtained using the seq2seq baseline.⁴¹⁷ The experimental results demonstrated the feasibility of transferring learning between models that operated with different chemical data sets.

Figure 16.

Illustration of transfer-learning-aided retrosynthetic analysis. To improve the accuracy of the antisynthetic analysis, a migration learning strategy in terms of the seq2seq and transformer models was employed. In this analysis, a large chemical reaction data set was pretrained to acquire specialized knowledge of chemical reactions. Such learned knowledge is then successfully transferred to a smaller data set. With the chemical information attained from the pretraining, the final model yields higher accuracy. Reproduced with permission from ref 416. Copyright 2020 Multidisciplinary Digital Publishing Institute.

In 2021, Chen et al.⁴¹⁸ presented a neural network-based predictor of log P, named MRLogP, for predicting the lipophilicity of small molecules using transfer learning techniques. MRLogP achieved an average RMSE of 0.988 and 0.715 when tested on druglike molecules from Reaxys and PHYSPROP, respectively. This work demonstrates that the application of transfer learning techniques enables accurate log P prediction even with small experimental training data sets.

In 2017, Cang and Wei built an algebraic topology-based multitask and multichannel CNN model for predicting protein stability changes upon mutations.⁴¹⁹ As shown in Figure 17, this model shared and transformed algebraic topological invariants for transfer learning to the impact of mutations on protein stability. The large globular protein data set of 2648 samples was shared and simultaneously trained with a small membrane protein data set of 223 samples, which improved the prediction correlation from 0.52 to 0.57. The performances also indicated that the proposed model holds significant potential for predicting protein–ligand binding affinities and mutation-induced protein stability changes.

Figure 17.

Schematic illustration of a multitask topological DL model.⁴¹⁹ Topological invariants extracted by element-specific persistent homology are shared among globular proteins and membrane proteins.

Transfer learning is an effective strategy for dealing with small data sets, as it can improve the accuracy of models for specific tasks. Transfer learning has been used in various fields, including activity prediction,⁴²⁰ protein domain,^421,422 drug prediction,^412,423,424 image classification,^425–427 text sentiment classification,^428–430 and multilingual text classification. Although transfer learning has been extensively applied in many fields, its application in small molecule data sets is still in the early stages. Further research is needed to investigate related theoretical aspects such as the issue of transferability and the importance of data similarity and task correlation in achieving success.

Additionally, quantifying the correlation between different tasks is a challenge for transfer learning. The migration performance may depend on the source and target tasks, where the correlation of the tasks is often more important than the size of the data. Transfer learning faces various challenges, such as transfer boundaries, even though it is mainly applied in small and less fluctuating data sets.⁴³¹ Effectively transferring knowledge from one task to another simply and clearly is a significant challenge. Furthermore, using the theory of transfer learning in multitask or multidomain situations is also a question, as there may be multiple domains that differ from the target domain. Although knowledge in multiple fields can be transferred, there may be problems in transferring multiple fields that need to be resolved. Overall, most transfer learning techniques that handle small training samples achieve good experimental results and thus the issues of efficiency, experimental cost, etc., should also be considered.

3.12. Active Learning

In the industrial and scientific communities, data must be annotated to be used in ML algorithms, but this process is typically time-consuming and expensive, in terms of human and material resources. To mitigate these costs, AL methods were proposed in the ML domain. The AL concept was first introduced by Lewis in 1994.⁴³² The basic idea is to iteratively query an information source to obtain desirable labels, which is also termed optimal experimental design.

In general, AL can be divided into two categories: stream-based AL^433,434 and pool-based AL.⁴³⁵ The stream-based AL framework requires all of the training data to be passed to the algorithm as a data stream. Each data point is sent to the algorithm separately for training, and the algorithm must decide immediately whether to label the data or not. Additionally, the training data are selected from the data pool for labeling, and the label of the current training data should be sent to the algorithm immediately before the next data point is trained. In contrast, the pool-based AL process is less complex than the stream-based approach. The training data come from an unlabeled data pool, and then the data are selected from the pool for labeling. Overall, AL is usually applied to scenarios with a large amount of unlabeled data in order to achieve the desired performance of the model with a few labeled samples.

Additionally, AL consists of five core components: the unlabeled pool, select queries, human annotator, labeled training set, and ML model. AL is mainly used in scenarios where data labeling is scarce or expensive, proactively requesting labeling and submitting the filtered data to experts for labeling to obtain a better model with fewer training samples. In the field of bioinformatics research, some chemical molecular data sets are typically small, making them ideal candidates for AL application. Recently, AL has played an essential role in predicting the biological and physical activities of small molecules in the fields of biology and chemistry. This includes predicting the structure of proteins,^436–438 as well as the toxicity of compounds.^439,440 However, how does AL deal with data sets that have a small number of labeled elements? Numerous works have been proposed to address this issue, which are outlined below.

In 2019, Zhang et al.⁴⁴¹ proposed a semi-supervised method using SVGD (stein variational gradient descent), called semi-supervised with SVGD, to quantify uncertainty in molecular properties. The method combined the algorithm SVGD⁴⁴² with semi-supervised learning and used AL to overcome the problem of data set bias in the training set, demonstrating that it can be robust to the uncertainty of molecules. The experiments used both small data sets, such as FreeSolv (sample size: 643), ESOL (sample size: 1128), and CatS (sample size: 595), and relatively large data sets, including MeltingPoint (sample size: 3025), p450 (sample size: 8817), and malaria (sample size: 13417). Two other methods, graph convolution with dropout and semi-supervised with dropout, were compared experimentally with semi-supervised with SVGD, where the first two methods were used in combination with dropout (dropout variational inference). The experiments were evaluated using the Spearman correlation coefficient, and the results are shown in Table 7. The experimental results demonstrated that semi-supervised with SVGD outperformed the other two methods on all six data sets.

Table 7.

Performance of the Three Models on Six Data Sets^a

data set	graph convolution with dropout	semi-supervised with dropout	semi-supervised with SVGD
FreeSolv	0.531 ± 0.061	0.439 ± 0.093	0.688 ± 0.053
ESOL	0.112 ± 0.035	0.306 ± 0.079	0.553 ± 0.026
CatS	0.049 ± 0.036	0.066 ± 0.044	0.310 ± 0.019
MeltingPoint	0.192 ± 0.016	0.284 ± 0.035	0.337 ± 0.013
p450	0.167 ± 0.015	0.185 ± 0.049	0.213 ± 0.010
malaria	0.315 ± 0.028	0.317 ± 0.031	0.378 ± 0.019

^aReproduced with permission from ref 441. Copyright 2019 Royal Society of Chemistry.

Peptides are a popular target for biomaterials design, and their data are often scarce. In 2021, Rainier et al.⁴⁴³ applied AL with CNN to binary classification of peptides, including two standard AL methods, query by committee and uncertainty minimization. The framework of the model is shown in Figure 18. The authors presented a multitask benchmark database of peptides designed to advance these methods for experimental design, and found that neither AL method tested to be better than random choice and combing meta-learning and AL could offer inconsistent benefits. Their findings validate that AL could be used an extension to design of experiments through the selection of optimal experiments on limited resources.

Figure 18.

Neural network structure for active learning. Here, $L_{\mathrm{m}\mathrm{a}\mathrm{x}}$ is the maximum width of a peptide in the data set (although a convolution can use any length), $K$ is the number of motif classes, and $A$ is the length of the amino acid alphabet. Peptides are first translated to a one-hot encoding $\left(L_{\mathrm{m}\mathrm{a}\mathrm{x}}\times A\right)$ and a vector of normalized amino acid counts $(1\times N)$. The output of the max pool layer is passed through one fully connected layer with ReLU activation, then, amino acid counts are appended to the output. This is then passed into two more fully connected layers with a final output dimension of 2 for positive and negative class labels. Labels below neural network layers indicate the dimensionality of the data as they pass through the layer. Reproduced with permission from ref 443. Copyright 2020 American Chemical Society.

In addition to the recent studies discussed above, AL has been applied not only in the field of biochemistry,⁴⁴⁴ but also in medical imaging,⁴⁴⁵ unmanned systems,⁴⁴⁶ and Internet big data,⁴⁴⁷ among others. AL has the potential to reduce the amount of annotated data required, as obtaining labeled data is a time-consuming and labor-intensive part of building ML models. As a relatively new ML method, AL aims to optimize operational resources and reduce the number of training samples.^448,449 The key idea of AL is to choose the appropriate annotation set and then manually annotate the data with the method selection depending on whether a single ML model or multiple ML models are used. Overall, AL strives to reduce annotation cost and increase model performance, enhancing the prosperity of applications in various scenarios, including imaging, NLP,⁴⁵⁰ safety risk control, and time series anomaly detection,⁴⁵¹ among others. AL has the potential to be applied to more scientific tasks in the future, and effective AL strategies for optimizing repeated training in continuous data acquisition remain an important research topic.

3.13. Graph-Based Semi-Supervised Learning

Traditional ML tasks can be broadly categorized into unsupervised and supervised learning. Semi-supervised learning is a hybrid approach that addresses learning tasks where only a portion of the data is labeled and the amount of labeled data is much smaller than the unlabeled data. This approach combines the strengths of both supervised and unsupervised learning. In many practical scenarios, manually labeling samples can be expensive, which leads to very sparse labeled data. However, unlabeled data are often easily obtainable. Semi-supervised learning leverages a large amount of unlabeled data along with a small amount of labeled data to train the model, thus addressing the problem of insufficiently labeled samples. For example, the Merriman–Bence–Osher (MBO) algorithm is a popular method used in semi-supervised learning tasks. The first step is to construct a graph with a specified number of nearest neighbors, denoted as Ne. Then, the Laplacian and a specified number of eigenvalues and eigenvectors, again denoted as Ne, are calculated from the graph. A subset of the input data is selected as the labeled set for training, while the remaining data are used as unlabeled data for testing. This approach has been shown to yield good results, particularly for small data sets in the fields of biology and chemistry. For instance, it can be applied to predict biochemical molecular interactions, such as interactions between proteins and drug molecules,^452,453 or interactions between proteins.^454,455 In addition, the Nyström technique enables MBO to be used effectively with very large data sets. This section outlines recently developed techniques for applying this method to ML prediction with small data sets.

In the field of molecular and biological sciences, small or insufficiently labeled data sets are a common challenge due to the high costs of experiments. In 2022, Hayes et al.⁴⁵⁶ proposed three new ML models, namely, an autoencoder coupled with an MBO scheme (AE-MBO), a bidirectional encoder transformer coupled with an MBO scheme (BT-MBO), and an ECFP⁴⁵⁷ coupled with an MBO scheme. The proposed models were validated with experiments on five data sets, and their performance was compared with other methods, such as SVMs, RFs, and gradient-boosted decision trees. Comparative experiments were conducted to test the effectiveness of the proposed models on a small amount of labeled data, using 1%, 2%, 5%, and 90% of labeled data from the data set in different models. The results for 1% labeled data for the five molecular classification data sets are presented in Figure 19. The proposed model in this article demonstrated strong predictive power in the presence of sparse marker data.

Figure 19.

Comparison of MBO-based proposed methods (shown in red) with other methods (shown in blue) on the five benchmark molecular data sets for 1% labeled data.⁴⁵⁶

In 2021, Merkurjev et al.⁴⁵⁸ proposed two MBO-based approaches for ML tasks with limited samples or small data sets. The first, called multikernel manifold learning (MML), integrated manifold learning with multikernel information. The second, called multiscale MBO (MMBO),⁴⁵⁹ introduced multiscale Laplacians to a modification of the MBO scheme. These approaches were tested on various types of data sets, including α and β-protein (sample size: 900). The experimental results demonstrated that the proposed MMBO method consistently outperformed other methods and emerged as the top performer in most experiments, with the MML method closely following. Recently, the Poisson equation was used for graph based semi-supervised learning at very low label rates.⁴⁶⁰ This approach replaced the assignment of label values at training points with the placement of sources and sinks in the Poisson equation. The resulting Poisson learning was compared with traditional Laplacian learning.

Semi-supervised learning has found wide application in various fields to solve problems encountered in real life. These fields include image classification,⁴⁶¹ sentiment analysis,⁴⁶² speech recognition,⁴⁶³ bioinformatics,^464,465 and many others. Classification-based semi-supervised learning methods are similar to supervised methods in that they require a large amount of training data to classify the test data and thus obtain a superior classification system.⁴⁶⁶

The field of semi-supervised learning is aimed at building efficient learning methods and improving learning performance by leveraging the information contained in unlabeled samples.⁵⁹ Semi-supervised clustering is a specific type of clustering that uses both labeled and unlabeled data with auxiliary information to help group data patterns.⁴⁶⁷ Additionally, reducing the dimensionality of high-dimensional data is a crucial technique in semi-supervised learning that often incorporates knowledge from the field of paired constraints.⁴⁶⁸ However, semi-supervised learning still poses a significant challenge, particularly in addressing the problem of lack of robustness. While increasing the amount of labeled data has been proposed to counter this issue, many algorithms used in semi-supervised scenarios struggle to obtain sufficient labeled data, making this a pressing open problem.

4. PERSPECTIVES FOR MOLECULAR SCIENCE

4.1. Combining Deep Learning with Traditional Machine Learning

DL is a crucial tool in the fields of computer vision,⁴⁶⁹ drug discovery,^470,471 and NLP,⁴⁷² where experiments often require a relatively large amount of data. DL has been widely used in the fields of chemistry and biology.^473,474 For instance, Yang and Li developed an interpretable uncertainty quantification method for DL-based molecular property prediction.⁴⁷⁵ Yang et al. explored the space of low-toxic chemicals through DL-based molecular generation.⁴⁷⁶ Additionally, Pandey et al. introduced state-of-the-art DL architectures for accelerating molecular docking, evaluating off-target effects, and predicting pharmacological properties.⁴⁷⁷

However, when using very small data sets, DL models may struggle to establish a reliable distribution or determine their coefficients, leading to high prediction errors despite good training performance. In contrast, traditional ML methods such as KNN,⁴⁷⁸ Bayesian network,⁴⁷⁹ SVM,⁴⁸⁰ and GBDT⁴⁸¹ tend to perform better on small data sets. While the DL is rapidly advancing, it is unlikely that it will fully replace ML algorithms. Instead, researchers have been exploring ways to combine the strengths of DL on large data sets with the strengths of ML on small data sets. This has led to an increasing amount of research on integrating DL with traditional ML algorithms for small data sets, as discussed below.

In 2022, Jiang et al. proposed a novel framework called boosting tree-assisted multitask DL (BTAMDL) for predicting chemical molecular properties.³⁴¹ The model consisted of multitask deep transfer learning and Gradient Boosting Decision Tree (GBDT). The BTAMDL model used small data sets in conjunction with related large data sets to learn the target and source tasks (involving small and large data sets, respectively) via multitask deep transfer learning and transferring knowledge from the source task to the target task. To validate the proposed method, the authors selected four types of data sets, including toxicity, log P, log S, and solvation. The toxicity data set consisted of four subsets with different sample sizes: LD₅₀ (7413), IGC₅₀ (1792), LC₅₀ (823), and LC₅₀-DM (353). The performance of BTAMDL was compared with that of other methods in the literature, and the results are presented in Table 8, where the first four methods are those involved in the paper, and the rest can be found in the literature. The results showed that the proposed BTAMDL framework can improve the prediction performance of small data sets.

Table 8.

Results of the R² Comparison of Accuracy between the Model in the Text and Other Methods for Molecular Property Predictions³⁴¹

method	IGC50	LC50	LC50-DM	average
BTAMDL 2	0.793	0.778	0.741	0.771
BTAMDL 1	0.795	0.776	0.733	0.768
MDL consensus	0.792	0.772	0.721	0.762
GBDT consensus	0.777	0.692	0.472	0.647
hierarchical482	0.719	0.710	0.695	0.708
single-model482	NA	0.704	0.697	0.701
FDA482	0.747	0.626	0.565	0.646
group contribution482	0.682	0.686	0.671	0.680
nearest neighbor482	0.600	0.667	0.733	0.667
test consensus482	0.764	0.728	0.739	0.744
3D MDL consensus338	0.802	0.789	0.678	0.765

In 2021, Qiu et al.⁴⁸³ presented a GBDT-based model called Bag-of-Words (BOW -GBDT for predicting drug interactions, as depicted in Figure 20. The framework consisted of three steps. First, features were obtained from the GPCR (G-proteincoupled receptor) module and combined with molecular fingerprint features. Second, the final features were generated through SMOTE (synthetic minority oversampling technique)⁴⁸⁴ and ANN. Finally, GBDT was used to predict drug interactions. The data set D92 M used in the study was a cross-validation data set with 1860 samples, and the Check390 data set was a test data set with 390 samples. The accuracy of BOW-GBDT was reported as 86.7%, which outperformed the accuracy of 82.8% achieved by the DWKNN (ensemble) method proposed by Xiao⁴⁸⁵ et al. in 2013.

Figure 20.

Illustration of a three-step process that can continuously select features to improve the accuracy of drug interactions during the experiment. In the first step, features are obtained through a GPCR module and merged with molecular fingerprints. Then, SMOTE (synthetic minority oversampling technique) and ANN are employed to generate the final features. Finally, GBDT is used to predict drug interactions. Reproduced with permission from ref 483. Copyright 2021 Frontiers Media SA.

In 2022, Yu et al.⁴² developed a powerful model called SVM +GCN, which used GCNs and SVMs to classify drug data sets. The data set used in the study was a small compound data set, and the SVM+GCN model was compared with SVM, RF, and GCN methods. Two validation methods, namely, random-split validation and fingerprint-split validation, were employed to evaluate the performance of the models. The results of the experiment showed that the SVM+GCN model achieved the highest accuracy at 95.8%, while the GCN and RF models obtained accuracies of 91.6% and 87.4%, respectively.

In 2021, Deng et al.⁴⁸⁶ proposed an integrated framework model, XGraphBoost, which combines the features of GNN with XGBoost⁴⁸⁷ for accurate molecular property prediction, as shown in Figure 21. The original molecular data was first formatted into a graph structure, and then the molecular features were extracted through GCN, GGNN (gated GNN), and directed message passing neural network. Finally, the XGBoost classifier was used to obtain accurate predictions of the molecular properties. The data sets used in the experiments were essentially small sample data sets, including ESOL (sample size: 1128), FreeSolv (sample size: 642), Lipophilicity (sample size: 4200), HIV (sample size: 41127), BACE (sample size: 1513), BBBP (sample size: 2039), Tox21 (sample size: 7831), ToxCast (sample size: 8575), SIDER (sample size: 1427), and Clintox (sample size: 1478). The experimental results demonstrated the advantages of combining the DL with traditional ML methods.

Figure 21.

Workflow of a joint GNN and XGBoost model. Molecular descriptors are extracted by a GNN model, and the prediction is produced by a supervised learner XGBoost for classification or regression. Reproduced with permission from ref 486. Copyright 2021 American Chemical Society.

The combination of DL methods and traditional ML algorithms is not only widely used in the field of biochemistry, but it is also prevalent in other research fields, such as cytotoxicity classification,⁴⁸⁸ disease research,⁴⁸⁹ and imaging.^490–492 While DL can improve prediction performance as the number of data increases, it falls short in its performance when small data sets are involved. Although DL has strong learning ability and portability, the number of model parameters will increase and their hardware requirements are also relatively high. Moreover, the model design can be relatively cumbersome. In contrast, the performance of traditional ML algorithms has an advantage in processing small data sets, and there is an increasing focus on developing methods that combine the advantages of the two. This direction represents an important area for future research.

4.2. Physical Model-Based Data Augmentation

The widespread use of ML and DL in fields such as imaging and text processing is heavily dependent on the quality and standardization of data. However, in the field of molecular science, due to the intricate complexity of molecular structures, especially macromolecules, it is challenging to obtain standardized features with the same dimensions. Many different molecule representations have been proposed,^339,493 and the field is still evolving. Adding to the challenge, there is often an insufficient amount of molecular samples available to build accurate and reliable ML models. To overcome this, researchers utilize traditional theoretical methods and physical models based on fundamental laws of physics to generate important parameters of molecular properties, which are used as labels to build and/or expand data sets. These labels are then used in downstream ML/DL procedures to predict molecular functions. Common theoretical methods and physical models used for this include molecular mechanics (MM),⁴⁹⁴ molecular dynamics (MD),^495,496 quantum mechanics (QM),⁴⁹⁷ quantum chemistry (QC),⁴⁹⁸ density functional theory (DFT),⁴⁹⁹ Monte Carlo method,⁵⁰⁰ and finite element analysis.⁵⁰¹ By using theoretical calculations and simulations, researchers can generate high-quality, diverse, and large-scale training data sets, which can significantly improve the predictive accuracy of ML models. For instance, to improve the accuracy of predicting chemical reactions for small data sets, physical theory and transfer learning can be employed, as shown in ref 502. In another study, Jian et al. extended the training data set by physically modeling T-cell receptors and peptide pairs.⁵⁰³ Additionally, Xie et al. presented an application of single-molecule ligation in monitoring molecular physical and chemical processes.⁵⁰⁴

MM, MD, and QM are all used to describe molecular interactions, including protein–protein, protein–nucleic acid, and protein–drug complexes.⁵⁰⁵ MD, which is based on force fields, can be used to depict conformational changes in proteins or nucleic acids, the impact of mutations on protein folding stability, and the binding energies between small molecules and proteins. QM, on the other hand, based on the Schrödinger equation, can predict electronic structures involved in chemical reactions and describe polarization effects, especially with dimension-reduced DFT. While the number of atoms in the system that can be handled by MM can be as large as a million, the system suitable for QM and DFT may have only a few dozen atoms. Therefore, QM/MM methods have been developed to combine the advantages of MM and QM while taking into account the computing cost of QM and the size of most biological systems.⁵⁰⁶ In QM/MM methods, the active site is handled by QM, while the rest of the system is considered by MM.

MM is a method used to calculate molecular structures and energies based on classical mechanical theory, using empirical and semiempirical parameters. The approach considers molecules as collections of atoms held together by elastic, van der Waals, and electrostatic forces, which reach equilibrium in the whole molecular system to determine its structure. The first use of MM dates back to 1927 when Born and Oppenheimer utilized it in their work, and it has since been widely used to calculate the conformations and energies of molecules. MM has been instrumental in determining and understanding the structure and properties of molecules since the 1950s.^507,508

MD is the most extensively researched method in MM. Its early simulations were focused on rigid spherical systems and gradually expanded to include molten salts, metals, alloys, semiconductors, and silicates.^509–513 Various useful algorithms, such as the truncation and modification algorithm of Lennard-Jones potential function, Coulomb interaction algorithm, Verlet nearest neighbor list algorithm, and lattice index algorithm, were developed during the evolution of MD simulations.^514–516 These algorithms have greatly influenced the application of MD simulations. In recent years, MM methods have expanded beyond the study of small- and medium-sized molecules and have become capable of handling macromolecular systems.⁵¹⁷ These methods are implemented in popular software packages, such as AMBER,⁵¹⁸ and have wide distributions and applications. In various fields, such as biophysics, biochemistry, coordination chemistry, materials, and physics,^519,520 as well as in drug design,^521,522 MM methods have been extensively utilized in conjunction with lattice dynamics, energy band theory, and many other approaches.

QM is a foundational theory that investigates the electronic structure and properties of atoms, molecules, condensed matter, atomic nuclei, and basic particles. One of the most popular QM methods used since the 1990s is DFT, which has become widely applied in the study of biomolecules and materials.^523–525 DFT provides accuracy levels similar to those of semiempirical methods but at a lower computational cost. It is commonly used in condensed matter physics, computational materials, and computational chemistry, and its high efficiency allows it to handle larger and more complex systems, expanding the range of applications and the predictive power of electronic structure theory. This has also fostered greater collaboration between modelers and experimentalists.⁵²⁶ However, due to the computational cost of QM methods and the large size of most biological systems, QM/MM methods have been developed to enable electronic structure calculations of biological systems.^518,527

As shown in Figure 22, Tavakoli et al.⁵²⁸ employed DFT to determine methyl cation affinities and methyl anion affinities for over 2400 organic molecules. This work resulted in a large data set of chemical reactivity scores, which is now available to the scientific community. The authors used this data set to train several DNN, each with different representations, in order to predict reactivity. Their findings revealed that graph attention neural networks outperformed other methods and representations, achieving a 10-fold cross-validation accuracy of 92%. This work highlights the power of combining QM and ML methods to enhance the scientific understanding and promote technological progress.

Figure 22.

Illustration of speed differences in computing a given molecular property for a database of 1 million molecules using DFT versus ML. On average, QM simulations require approximately 5 h per molecular structure, leading to a total processing time of 5 × 3600(s) × 10⁶ ≈ 500 years. In contrast, a trained DL model needs only 5 ms per molecular structure and just a few hours for 1 million molecules. Reproduced with permission from ref 528. Copyright 2022 American Chemical Society.

Qiao et al.⁵²⁹ proposed OrbNet, a framework that combined symmetry-adapted atomic orbitals features with a GNN to predict energy solutions. The experimental flowchart of OrbNet is presented in Figure 23. The authors demonstrated that OrbNet achieved prediction accuracy similar to that of DFT, but at a computational cost at least 3 orders of magnitude lower than DFT. OrbNet has been trained on approximately 100 000 molecules, and the training set can be further expanded to include more data.

Figure 23.

Illustration of the ORBNET workflow. (a) A low-cost mean-field electronic structure calculation is performed for the molecular system, and (b) the resulting SAAOs and the associated quantum operators are constructed. (c) An attributed graph representation is built with node and edge attributes corresponding to the diagonal and off-diagonal elements of the SAAO tensors. (d) The attributed graph is processed by the embedding layer and message-passing layers to produce transformed node and edge attributes. (e) The transformed node attributes for the encoding layer and each message passing layer are extracted and (f) passed to MPL-specific decoding networks. (g) The node-resolved energy contributions ${\epsilon }_{\mathrm{u}}$ are obtained by summing the decoding networks outputs nodewise, and (h) the final extensive energy prediction is obtained from a one-body summation over the nodes. Reproduced with permission from ref 529. Copyright 2020 AIP Publishing.

Bennett et al.⁵³⁰ developed 3D-CNN and spatial graph CNN models using atomic and molecular features based on atomistic MD simulations that calculated transfer free energies of 15 000 small molecules from water to cyclohexane. The DL models were trained to predict the transfer free energies based on MD-simulated data. The spatial graph CNN model showed higher accuracy than the 3D-CNN model, achieving a MAE of 4 kJ/mol when compared with MD calculations. This study suggests that the DL model can be a cost-effective alternative to expensive free energy calculations while providing similar accuracy to MD calculations. The experimental workflow is presented in Figure 24.

Figure 24.

Illustration of a deep CNN. The relative free energy for moving 15 000 small molecules between water and cyclohexane was computed with atomistic MD simulations. From the simulations, features, such as each atom’s partial charge, the average number of water contacts, and molecular features, including the number of hydrogen bonds and size/shape, were extracted. A 3D-CNN and spatial graph CNN were then constructed using the atomic and molecular features to predict the free energies of transfer. Reproduced with permission from ref 530. Copyright 2020 American Chemical Society.

In the field of molecular activity prediction research, combining two-dimensional or three-dimensional descriptors with ML, can be effective for identifying active compounds. However, training ML models on data generated by MD is still being explored. In 2019, Jamal et al.⁵³¹ obtained MD descriptors using simulations and combined them with 2D and 3D descriptors. They conducted experiments using two models: ANN and RF. The final results showed that the MD descriptor outperformed both the 2D and 3D descriptors, indicating a significant improvement in the classification performance of the obtained MD descriptor.

During computer-assisted drug design, the quantitative structure/property relationships model combines experimental descriptors with those generated by MD or QM to expand data sets, which improves the prediction of molecular properties. However, the applicable conditions of each computational simulation method are limited. For example, DFT can simulate only small molecular systems, and errors in the simulation structure under high-temperature, high-pressure, and strong magnetic field environments are often significant. In addition, MD simulations are usually dependent on the accuracy of the potential function. For some applications, such as inferring force fields by ML, access to a large and diverse high-quality training data set obtained from QM calculations is essential to capture reliable results for general applications.⁵³² However, there are still no known criteria for sufficiency, such as the question of how many molecular descriptors are required to explain ligand binding satisfactorily, or the question of how many large noncoding RNAs are diverse enough to represent the universe of RNA folds for these systems.⁵³³

Combining the simulation of these physical models with various ML algorithms could benefit the improvement of the QSAR model.⁵³⁴

4.3. Spatial and Temporal Pattern Extractions for Molecules

In recent times, there has been a significant increase in the availability of spatio-temporal data. Spatial pattern extraction, as demonstrated in ref 535, is commonly utilized to identify patterns or relationships in data that are associated with the spatial arrangement or position of data points. This is particularly useful in image classification or object detection tasks, where the accurate prediction of spatial relationships between pixels or points is crucial. Similarly, temporal pattern extraction is applied to identify patterns or relationships in data that are related to the sequence or timing of the data points. This technique is often employed in speech recognition⁵³⁶ or NLP,⁵³⁷ where the order of words or sequence holds significant importance for the data. With the development of molecular science, there has been growing interest in applying spatial and temporal pattern extraction to chemical and biological molecules. ML algorithms have made significant contributions to this field, as evidenced by numerous achievements.⁵³⁸ In the study by Roth et al.,⁵³⁹ it was found that material platforms like nanoparticles, hydrogels, and microneedles can be designed to control the interaction of vaccine components with immune cells spatially and temporally. Similarly, Goel et al.⁵⁴⁰ explored an avenue to go beyond the space of known drug-like chemistry to benefit drug design.

A wide range of ML algorithms are available for spatial and temporal pattern extraction, including CNNs, RNNs, LSTM, GraphCNN, Autoencoders (AEs)/Stacked Autoencoders (SAEs), and Sequence-to-Sequence (Seq2Seq) models.

CNNs are primarily used to process spatial maps and are often applied to tasks such as image classification and object detection, as demonstrated in ref 144. GraphCNN is designed to handle graph data and can be further categorized into spatial maps, as shown in ref 541. RNN models, including LSTM and GRU, are particularly effective in dealing with trajectories, time series, and the sequences of spatial maps, as discussed in refs 542 and 543. ConvLSTM, a hybrid model that combines RNN and CNN, is typically used for handling spatial maps, as described in ref 544. AEs and SAEs are well-suited for extracting features from time series, trajectories, and spatial maps, as detailed in refs 308 and 545. Lastly, Seq2Seq models are generally designed for sequential data and are used for cases involving time series and trajectories, as explained in ref 416.

4.4. Natural Language Processing for Molecular Sequences

NLP, as discussed in ref 546, is the ability of a computer program to understand, interpret, and generate human language, both spoken and written, which is known as natural language. As a component of AI, NLP has various real-world applications such as language translation,⁵⁴⁶ text classification,⁵⁴⁷ text generation,⁵⁴⁸ spam detection,⁵⁴⁹ virtual agents and chatbots,^546,550 and social media sentiment analysis.⁵⁵¹ Now, with more research, NLP is also being applied to chemical and biological molecules, showing powerful effects.^552,553

NLP is also being increasingly applied to chemical and biological molecules, with promising results.^552,553 For example, Winter et al. used NLP to predict limiting activity coefficients from SMILES codes.⁵⁵⁴ They developed a SMILES-to-properties transformer, an NLP network that accurately predicted binary limit activity coefficients from SMILES codes alone. Similarly, Lu and Zhang developed a unified DL model called T5Chem that used the Text-to-Text Transfer Transformer (T5) framework in NLP to predict various chemical reaction tasks.⁵⁵⁵ They found that models trained with multiple tasks were more robust and can benefit from the mutual learning of related tasks. In addition, NLP can be used to predict the physiological effects of chemicals, as demonstrated by Mukherjee et al. who developed models for predicting physiological effects of chemicals based on their molecular structures using NLP methods.⁵⁵⁶ They achieved high prediction accuracy using standard chemical data sets.

The process of NLP, as described in ref ⁵⁴⁶, can be divided into two main steps: data preprocessing and algorithm development. Data preprocessing involves preparing and cleaning text data, putting it in a workable form, and highlighting features in the text that can be analyzed by an ML algorithm. After the data have been preprocessed, an algorithm is developed to process it. There are two main types of algorithms: rules-based and ML-based. Rules-based algorithms are early NLP algorithms that use designed linguistic rules, while ML-based algorithms are used for tasks based on fed training data and can adjust their methods as more data is processed. There are various ML-based algorithms that can be used for NLP tasks, including the BOW algorithm, N-gram algorithm, word-embedding algorithm, RNN, and transformers, as explained in refs ^557–561.

Syntax and semantic analysis are two primary techniques used in NLP, as explained in ref ⁵⁶². Syntax refers to the arrangement of words in a sentence to create grammatical sense, and NLP utilizes syntax to extract meaning from language based on grammatical rules. On the other hand, semantics is concerned with the meaning behind words. NLP uses various algorithms to comprehend the meaning and structure of sentences.

4.5. Generative AI for Molecular Generation

Generative AI or generative models⁵⁶³ are a branch of unsupervised learning techniques in ML that are able to generate new data samples similar to a training data set, which are often used for tasks such as image generation, text generation, and data augmentation. They can also be effective in cases such as anomaly detection, where the goal is to identify examples that do not fit with the rest of the data. Generative networks or generative models are becoming increasingly popular in the field of chemical and biological molecules. According to Bilodeau et al., generative models can offer a new approach to molecular discovery by reframing molecular design as an inverse design problem.⁵⁶⁴ Similarly, Tong et al. stated that generative models have received a lot of attention in recent years, with researchers applying them to new drug design.⁵⁶⁵ They listed a number of publicly available generative-model-based molecular design tools that can be used to directly generate molecules. Additionally, in the study by Yakubovich et al., a computational workflow based on quantum chemical calculations and a DNN-based generative model was proposed for the discovery of novel materials.⁵⁶⁶ Here are four popular examples of generative network approaches as following.

GANs treat the training process as a game between two separate neural networks: a generator network and a discriminative network.²⁹⁹ The generator network is trained to generate new data samples, however, the discriminative network is trained to classify samples as either coming from the true distribution or the model distribution. Every time the discriminator notices a difference between the two distributions and the generator adjusts its parameters slightly to make it go away, until at the end, the generator exactly reproduces the true data distribution and the discriminator cannot find a difference between the two distributions.

Variational autoencoders (VAEs)⁵⁶⁷ are neural networks designed to learn an identity function in an unsupervised way to reconstruct the original input while compressing the data in the process so as to discover a more efficient and compressed representation. VAEs usually consist of an encoder network and a decoder network. The encoder network is trained to map input data samples to a latent space, while the decoder network is used to map points in the latent space back to the original data space. VAEs can be utilized to generate new data samples by sampling points in the latent space and passing them through the decoder network.

In drug discovery, it remains a challenge to create novel compounds that are not only druggable but also cheaply available. Gao et al.⁵⁶⁸ proposed a generative network complex (GNC) model to enable the design of optimal lead compounds with desired chemical properties. The framework of the GNC model is shown in Figure 25 and GNC generated new drug-like molecules based on the multiproperty optimization in the latent space of an autoencoder. Both Monte Carlo-like random diffusion algorithm and gradient descent were used to create new molecules in the latent space. The resulting compounds were translated into SMILES strings by a decoder and further evaluated by the real space ML models.

Figure 25.

Schematic illustration of a generative network complex for molecular generation.⁵⁶⁸

Autoregressive models like PixelRNN⁵⁶⁹ generate new data samples by predicting each data point in the sample based on the previous data points, which are commonly used for cases such as language modeling with the goal of predicting the next word in a sentence based on the previous words.

Generative pretraining (GPT)⁵⁷⁰ is one of the pioneers in language understanding and modeling, and essentially proposes the concept of pretraining a language model on a huge corpus of data, and then fine-tuning the model for downstream tasks. The core ideas of GPT are attention mechanism and unsupervised pretraining. The reason for unsupervised learning is the shortage of massive labeled data sets. GPT and its extensions GPT-2 and GPT-3 are well-known for their impressive performance on small data or zero-shot learning which is a scenario wherein at test time the samples provided are not observed while training, and have successfully applied to a variety of tasks, such as machine translation,⁵⁷¹ question-answering,⁵⁷⁰ reading of conceptual works, scripting of poems and elementary mathematics, etc., ChatGPT has gained a lot of popularity recently due to its impressive strengths, such as increased efficiency and precision in NLP-related tasks. It is capable of providing answers to a wide range of issues promptly and accurately, making it invaluable in assisting with routine tasks, generating algorithms for computing tasks, and much more. However, its potential applications in molecular science, especially, in small molecular data sets, have yet to be fully explored because ChatGPT is trained on an extensive corpus of data. It is likely that ChatGPT will be proven useful in studying chemical and biological molecules in the future, but further research is still needed to confirm this.

4.6. Material Science

In recent years, machine learning methods have been successfully applied to predict chemical and material properties, particularly in material science. However, due to restrictions or limitations, collecting large labeled training samples is typically difficult in this field, which significantly reduces the predictive power of sophisticated deep learning models like convolutional neural networks and recurrent neural network. To address these small data challenges, simple regression models can be used by creating linear combinations of nonlinear basis functions.⁵⁸⁰ For instance, when predicting the properties of elpasolite crystals, deep learning with a black box model may not be the optimal option for exploring the elpasolite universe and predicting the spin states of transition metal complexes. In such cases, the nature of the variables present in the linear model and the knowledge of the physics of the underlying problem can facilitate the identification of when simplistic linear solutions will bring comparable performance. Linear solutions can not only accurately predict material properties such as the bandgap and formation energy of transparent conducting oxides, the spin states for transition metal complexes, and the formation energy for elpasolite structures but also offer an excellent approach for interpretable predictions in the material science community.

5. OUTLOOK

In this review, we examine recent progress in addressing the challenge of working with small scientific data sets in machine learning and deep learning. Due to various constraints and limitations in data acquisition, small data sets are ubiquitous in scientific fields. The small data challenge in machine learning can be just as severe, if not more so, as the big data challenge. One of the most immediate problems posed by small scientific data sets is overfitting, which can occur not only during training but also during testing, ultimately leading to less accurate and reliable machine learning models. Additionally, small data sets are often associated with data imbalance. For example, in drug discovery, only a few drug candidates may be active, whereas for machine learning modeling, active and inactive samples should be well-balanced. Data imbalance can result in inaccurate, unreliable, and unstable machine learning and deep learning models. Moreover, augmenting small data sets using computational approaches can easily introduce noise or nonuniform data, which also presents a challenging issue in machine learning and deep learning. As summarized in Table 9, this paper reviews several approaches to address the challenges posed by small data sets, including transfer learning or multitask learning, combining traditional ML algorithms with deep learning, self-supervised learning, Generative Adversarial Networks, variational autoencoders, transformers, long short-term memory, active learning, semi-supervised learning, and physical model-based data augmentation. While many of these approaches have been proposed in the past decade and are still in the early stages of development, there have been tremendous advances in recent years. However, the small data challenge remains a pressing issue in machine learning and deep learning, calling for innovative strategies.

Table 9.

An Overview of Major Machine Learning and Deep Learning Approaches in Different Fields with a Variety of Algorithms for Small Data Challenge

applied field	algorithm	ref
Basic Machine Learning Algorithm Approach
drug–target interaction	gradient boosted decision trees (GBDT)	40
drug-induced ototoxicity	support vector machine (SVM), message-passing neural networks (MPNNs)	111
compound activity	random forest (RF), k-nearest neighbor (KNN)	114
miRNA expression	random forest (RF)	115
Artificial Neural Networks Approach
molecule lipophilicity	MRlogP, a neural network-based predictor of log P	131
molecule lipophilicity	multiple linear regression (MLR) and artificial neural network (ANN)	132
molecule lipophilicity	GA-MLR and GA-ANN	133
pharmacokinetics	quantitative structure–pharmacokinetic relationship model	134
dislocation nucleation	artificial neural networks (ANNs), random forest (RF), support vector machine (SVM)	135
molecular dynamics simulations	k-nearest neighbor (k-NN) and artificial neural network (ANN)	136
Convolutional Neural Networks Approach
drug-induced liver injury	natural language processing (NLP) inspired convolutional neural networks (CNNs)	168
environmental applications	molecular image-convolutional neural networks (CNNs) with transfer learning	169
molecular dynamics simulations	deep learning encoder–decoder convolutional neural networks (CNNs)	172
U-Net Approach
binding sites prediction	Voxel-based 3D U-Net	200
protein structure prediction	single-sequence-based U-Net convolutional network	201
medical image segmentation	an automatic liver parenchyma segmentation network based on the U-Net architecture	202
Graph Neural Networks Approach
molecular property prediction	property-aware relation networks with graph neural networks-based classifier	229
machine learning algorithm	model agnostic meta-learning (MAML) and first-order MAML (FO-MAML)	231
molecular property prediction	meta-MGNN	232
Long Short-Term Memory Approach
protein structure prediction	deep asymmetric convolutional LSTM neural network (DeepACLSTM)	258
medicinal science	deep learning long short-term memory (DL-LSTM)	263
protein remote homology detection	ProDec-BLSTM	264
anticancer peptide prediction	bidirectional long short-term memory (BLSTM)	272
short-term load forecasting	bidirectional LSTM	273
Generative Adversarial Networks Approach
protein solubility prediction	protein log S generative adversarial nets (Pro-GAN)	281
multiclassification for cancer staging	Generative Adversarial Network (GAN) combined with a deep neural network (DNN)	284
Generative Adversarial Networks Approach
cancer classification	Gene-GAN Wasserstein generative adversarial	74
cancer prognosis prediction	network-based deep adversarial data augmentation (wDADA)	289
brain network	BrainNetGAN	290
antiviral drug	sequence-based binary classifier	291
Autoencoders Approach
materials science	variational autoencoder (single task learning) (VAE), VAE-L with linear regression (multitask learning), VAE-NL with nonlinear regression (multitask learning)	64
biopolymerization	VAE+ANN, VAE+GAN, VAE+RF, GAN+RF, ANN, RF	41
drug discovery	autoencoder-assisted multitask ANN	315
Transformers Approach
drug discovery	self-supervised Motif Learning Graph Neural Network (MoLGNN)	335
molecular property prediction	algebraic graph-assisted bidirectional transformer	334
drug repositioning	multitask self-supervised learning	344
Reinforcement Learning Approach
molecular property prediction	deep reinforcement learning with new reward function	372
human microRNA-disease association	RFLMDA (combining the Q-learning algorithm with reinforcement learning)	373
molecule design	deep reinforcement learning	378
cancer type classification	deep reinforcement learning	379
Transfer Learning Approach
pharmacokinetic parameter prediction	integrated transfer learning and multitask learning approach	412
pharmacogenomics	adversarial inductive transfer learning (AITL)	413
retrosynthetic analysis	sequence-to-sequence (seq2seq) transfer learning	416
chemical reaction prediction	transformer-transfer learning	572
molecular property prediction	MRlogP (neural network-based predictor of log P)	418
protein structure prediction	algebraic topology-based multitask and multichannel CNN	419
Active Learning Approach
domain applicability	semi-supervised learning and Bayesian deep learning	441
molecular property prediction	active learning -based semi-supervised GNN	573
Graph-Based Semi-supervised Learning Approach
graph-based semi-supervised learning	an autoencoder coupled with Merriman Bence Oshe scheme (AE-MBO) and a bidirectional encoder transformer coupled with Merriman Bence Oshe scheme (BT-MBO)	456
machine learning algorithm	Multikernel manifold learning (MML) and multiscale MBO (MMBO)	458
machine learning algorithm	Poisson Merriman Bence Oshe (MBO)	460

Given the widespread need for machine learning techniques to handle large-scale training samples coupled with the increasing progress of small data studies, the concepts and methods of small data research are now being applied to diverse applications. In this regard, we highlight a few forefronts of the development and application of the machine learning methods for small data challenges in molecular science, particularly in molecular properties discovery, multilinear models in material science, machine learning force fields,^574,575 protein folding,⁵⁷⁶ catalyst design,⁵⁷⁷ and retrosynthetic pathways.⁵⁷⁸

5.1. Machine Learning Force Fields

Machine learning force fields^574,579 are applied to overcome the size limitations of accurate ab initio methods, by learning the energies and interactions in atomic-scale systems directly from, for example, density functional theory calculations. Unlike conventional force fields, Machine learning force fields are built on mathematical structures with limited underlying physics and chemistry concepts. Therefore, it is crucial to train the machine learning force field on relevant density functional theory data, such as energies, forces, and stress, to obtain a robust Machine learning force field for particular systems and applications. During training, the atomic environments in a configuration are transformed into a set of features that are then used to predict the energies of the atomic configuration for downstream tasks. Once the training is complete, the machine learning force field model can be used for atomic-scale simulations, much like any other conventional force field.

5.2. Biomolecular Properties Discovery

One of the most challenging issues in drug design and substance discovery is predicting molecular properties. Traditional methods based on density functional theory have explicit physical images but are time-consuming for processing large numbers of molecules. In recent years, data-driven machine learning models have successfully learned the relationship between the structure and properties of a molecule and can perform low-cost predictions instead of costly and time-consuming processes involving human expertise, computer simulation, and subsequent experimental synthesis. However, due to the complexity, cost, and time required to obtain molecular information experimentally, it is often difficult to obtain large labeled molecular data sets. Several approaches have been developed to address this challenge. Hayes et al. introduced three graph-based MBO models for molecular classification prediction with scarcely labeled data, including Ames, Bace, BBBP, Beet, and ClinTox data sets.⁴⁵⁶ Jiang et al. built a BTAMDL architecture that integrates GBDT and multitask deep learning to achieve near-optimal predictions for small molecular properties such as partition coefficient, solubility, toxicity, and solvation.³⁴¹

5.3. Protein Folding Prediction

Protein folding plays a decisive role in the biological functions of proteins. Predicting protein folding modes is crucial in expressing their spatial topological features and can be solved as a classification problem with ML methods. Generally, machine learning algorithms take amino acid sequences as input and predict folding patterns by extracting features, which are then fed into a classifier for prediction and performance evaluation. Alphafold2 is currently the most popular tool for protein folding, which combines knowledge of protein structure with deep learning.⁵⁸¹ While Alphafold2 has achieved significant success in protein folding prediction, its predictive accuracy is lower compared to experimental techniques such as X-ray crystallography.⁵⁸² Additionally, running Alphafold2 requires substantial computational resources. Other advanced methods such as DeepSVM fold have also been proposed, which achieved a prediction accuracy of 67.3% and outperformed other methods.⁵⁸³

5.4. Catalyst Design

To understand catalyst catalysis, it is necessary to accurately identify descriptors of catalytic activity.⁵⁸⁴ However, traditional methods often lack predictability and accuracy, leading researchers in catalyst design to focus on improving the accuracy of identifying catalyst descriptors and predicting rates using machine learning. In a recent study by Wenjie Liao et al.,⁵⁸⁵ an enhanced method for accurately identifying descriptors was proposed using a machine learning surrogate model derived from a kinetic data set, which outperformed traditional derivative-based methods. Density functional theory is a commonly used computational chemistry tool for studying and predicting the geometric structure, mechanical properties, electronic structure, and reaction energies of materials. Xuhao Wan et al.⁵⁸⁶ introduced a DFT-based machine learning approach (DMCP) and used transition metal phthalocyanine diatomic catalysts as electrocatalysts for carbon reduction reactions.

5.5. Retrosynthetic Pathways

Retrosynthesis, which was proposed by Corey in the 1960s, describes the iterative process of reducing a complex target molecule to a simple precursor by breaking bonds.⁵⁸⁷ It summarizes the reverse work that organic chemists need to do when building new molecules, and these chemists have identified a series of chemical transformations that can be achieved through the simpler chemical structure of oil or other resources.⁵⁸⁸ Currently, retrosynthetic programs are mainly divided into logic-based heuristic programs and detailed retrosynthetic route prediction programs.⁵⁸⁹ Moreover, Badowski et al.⁵⁹⁰ have shown that synergy between expert and machine learning approaches can lead to improved retrosynthetic planning. In the future, high-quality databases will accelerate further developments in retrosynthesis.⁵⁹¹

5.6. Computational Chemistry

Recently, computational chemistry and machine learning have increasingly been combined to enhance the understanding and prediction of chemical and physical properties and behavior. More and more machine learning and deep learning techniques are borrowed in computational chemistry to generate models and algorithms to extract patterns and relationships from large data sets and to make predictions about chemical systems. For some chemical systems with limited data, there has been great progress made in recent years. For instance, Lilienfeld et al. developed quantum machine learning models to predict various molecular properties, such as energy, electronic structure, and spectroscopic data.⁵⁹² Ceriotti et al. explored different machine learning methods to understand the behavior of molecules and materials in systems with various size at the atomic scale.⁵⁹³ Moreover, Csanyi et al. provided a unified framework to predict atomic-scale properties based on local description of chemical environments and Bayesian statistical learning.⁵⁹⁴ More related references can be found in refs 595–600.

We conclude our review with several reminders of challenges that need to be addressed when dealing with small data sets in machine learning and deep learning.

5.7. Modelability Metrics

It is essential to develop metrics for measuring the modelability of small data sets, which can be used to evaluate all methods, including transfer learning, where the data similarity index is closely related to the modelability.

5.8. Small and Diverse Data Sets

Developing machine learning and deep learning methods for handling small and diverse data sets is particularly challenging. Data diversity is closely related to data modelability, especially for small data sets.

5.9. Small and High-Dimensional Data Sets

Developing machine learning/deep learning methods for tackling small and high-dimensional data sets, especially for single-cell RNA sequencing (sc-RNA-seq) and transcriptomic data analysis, is another important task. Traditional approaches such as principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and uniform manifold approximation and projection (UMAP) have limited success, and machine learning and deep learning methods are expected to address challenges from the spatiotemporal entanglement of cells, genes, and tissues.

5.10. Small and Noisy Data Sets

Moreover, addressing truly small and noisy data sets is one of the most challenging tasks in machine learning and deep learning. Currently, there is limited feasibility and few results for this problem in the literature.

5.11. Small and Imbalanced Data Sets

The modeling of small and imbalanced data is a difficult issue that needs to be addressed. Imbalanced data sets naturally occur in experimental settings where successful results are reported while unsuccessful ones are ignored. On the other hand, in drug discovery, most drug candidates are unsuccessful.

5.12. Data Imputation in Small Data Sets

Treating concurrent small data and data imputation can be very challenging. This treatment is often a needed preprocessing in machine learning studies. It will be an important research topic.

5.13. Data Representability

The quantitative analysis of data representability will be an interesting issue. The construction of effective descriptors will continue to be an important area of research, particularly for data with intrinsically complex internal structures, such as biomolecules, macromolecules, and functional materials.

5.14. Machine/Deep Learning Complexes

The construction of sophisticated machine learning complexes that integrate different ML methods to deal with small data sets, such as using migration learning in combination with Generative Adversarial Networks, while optimizing the data and the model framework to obtain the desired results, will be both challenging and important. It is expected that such complexes will become common in molecular sciences.

5.15. Data Understanding

Finally, one cannot overemphasize the role of physical/chemical/biological understanding of data in the machine learning method design, development, or selection and machine learning result interpretation. It is important to utilize prior domain knowledge about small data sets to design machine learning and deep learning methods and improve their predictability.

We apologize that we could not review all related concepts and issues and cover all important references about machine learning and deep learning approaches for dealing with small scientific data sets. We hope that the reader can benefit from the perspective presented in this review and find a way to tackle the small data challenge.

ABBREVIATIONS

AI

artificial intelligence

AL

active learning

ANN

artificial neural network

AUC

area under the curve

BLSTM

bidirectional LSTM

BOW

bag-of-words

BTAMDL

boosting tree-assisted multitask deep learning

CNNs

convolutional neural networks

DFT

density functional theory

DL

deep learning

DNN

deep neural networks

Deep RL

DRL

ECFP

extended-connectivity fingerprint

F-measure

F-measure represents the harmonic mean of precision and recall

GAN

Generative Adversarial Network

GBDT

Gradient Boosting Decision Tree

GF-VAE

Graph Flow-Variational AutoEncoder

GNC

generative network complex

GNN

graph neural network

GCNs

graph convolutional networks

GPCR

G protein-coupled receptor

KDE

kernel density estimation using Gaussian kernel function

KNN

K-nearest neighbor

LSTM

long short-term memory

log P

partition coefficient

log S

solubility

MBO

Merriman–Bence–Osher

MAE

mean absolute error

MD

molecular dynamics

ML

machine learning

MM

molecular mechanics

MMBO

multiscale MBO

MML

multikernel manifold learning

MoLGNN

Motif Learning Graph Neural Network

NB

naive Bayes

NLP

natural language processing

ProGAN

Protein Solubility Generative Adversarial Net

QC

quantum chemistry

QM

quantum mechanics

RF

random forest

RNNs

recurrent neural networks

R²

coefficient of determination

RMSE

root mean square error

SSL

self-supervised learning

SVM

support vector machine

VAE

variational auto-encoder

Biographies

Biographies

Bozheng Dou obtained his B.S. degree in Geomatics Engineering in 2021 from Nanjing University of Information Science and Technology. He is currently a M.S. candidate at Wuhan Textile University under Dr. Jie Liu and Dr. Jian Jiang.

Zailiang Zhu obtained his B.S. degree in Software Engineering in 2019 from Zhongyuan University of Technology and received his M.S. degree in Software Engineering in 2023 from Wuhan Textile University under Dr. Bengong Zhang and Dr. Jian Jiang.

Ekaterina Merkurjev is an Assistant Professor in the Departments of Mathematics and CMSE at Michigan State University. She received her PhD in Applied Mathematics from the University of California, Los Angeles, in 2015 under the supervision of Prof. Andrea Bertozzi. She also obtained her Bachelors degree from the University of California, Los Angeles. Her research interests include graph-based methods, semi-supervised learning, and image processing.

Lu Ke obtained her B.S. degree in Applied Mathematics in 2022 from Wuhan Textile University. She is currently a M.S. candidate at Wuhan Textile University under Dr. Bengong Zhang and Dr. Jian Jiang.

Long Chen obtained his B.S. degree in Information and Computing Science in 2022 from Wuhan Textile University. He is currently a M.S. candidate at Wuhan Textile University under Dr. Yueying Zhu and Dr. Jian Jiang.

Jian Jiang received his B.S and M.S. degrees in Theoretical Physics at Central China Normal University in 2005 and 2007, respectively. He received his Ph.D. degree in Theoretical Physics from University of Le Mans in France in 2011. He is currently a professor in the School of Mathematical and Physical at Wuhan Textile University in China. His research interest includes topological data analysis on molecular science, drug design and discovery, data mining, and modelling and analysis of complex networks.

Yueying Zhu obtained her Ph.D. degree in Physics from Le Mans University and Central China Normal University under the mentorship of Profs. Qiuping Alexandre Wang, Xu Cai, and Wei Li. Her Ph.D. study focused on the uncertainty and sensitivity analysis of nonlinear dynamical systems, and the modeling and simulations of spreading dynamics on complex network. Now, she is an associate professor in Prof. Jie Liu’s group at Wuhan Textile University. Her current research concerns the application of uncertainty and sensitivity analysis to spreading dynamics, especially epidemic and opinion spreading on a complex network.

Jie Liu received his B.S. degree in Mathematics from Hubei Normal University, China, in 1997, and received his M.S. degree and Ph.D. degree in Application Mathematics and Computation Mathematics from Wuhan University, China, in 2003 and 2006, respectively. He is currently a professor in School of Mathematical and Physical at Wuhan Textile University in China. His research interest includes the analysis of dynamical nonlinear systems, control of complex systems, and modelling and analysis of complex networks. He is the author of more than 80 journal and conference papers.

Bengong Zhang received his Ph.D. degree in Applied Mathematics from South China University of Technology in 2010 and completed his postdoctoral studies at The University of Tokyo in 2013 under the guidance of Prof. Kazuyuki Aihara and Luonan Chen. His postdoctoral studies focused on computational systems biology. Now he is the professor of Wuhan Textile University, Wuhan, China. His current research concerns scRNA-seq data analysis and machine learning.

Guo-Wei Wei earned his Ph.D. degree from the University of British Columbia in 1996. He was awarded a fellowship from the NSERC of Canada to pursue his postdoctoral work at the University of Houston. In 1998, he joined the faculty of the National University of Singapore and was promoted to Associate Professor in 2001. In 2002, he relocated to Michigan State University, where he is an MSU Foundation Professor of Mathematics, Electrical and Computer Engineering, and Biochemistry and Molecular Biology. His research explores the mathematical foundations of biological science and data science, including deep learning, drug discovery, and computational geometry, topology, and algebra. Dr. Wei has served extensively on a wide variety of national and international panels, committees, and journal editorships.