Structural and functional prediction, evaluation, and validation in the post-sequencing era

Abstract

The surge of genome sequencing data has underlined substantial genetic variants of uncertain significance (VUS). The decryption of VUS discovered by sequencing poses a major challenge in the post-sequencing era. Although experimental assays have progressed in classifying VUS, only a tiny fraction of the human genes have been explored experimentally. Thus, it is urgently needed to generate state-of-the-art functional predictors of VUS in silico. Artificial intelligence (AI) is an invaluable tool to assist in the identification of VUS with high efficiency and accuracy. An increasing number of studies indicate that AI has brought an exciting acceleration in the interpretation of VUS, and our group has already used AI to develop protein structure-based prediction models. In this review, we provide an overview of the previous research on AI-based prediction of missense variants, and elucidate the challenges and opportunities for protein structure-based variant prediction in the post-sequencing era.

1. Introduction

With the surge of high-throughput sequencing technologies, a number of large-scale genetic projects have been launched by several countries[1]. The exponentially increasing genome sequencing data have revealed that there are extensive genetic variations within human populations[2], [3]. The vast majority of these variants are defined as variants of uncertain significance (VUS) because their phenotypic consequences and clinical significance are unknown. Therefore, in the post-sequencing era, the main task of genomic research has changed to converting the rich genetic data into useful information. This is a particularly acute problem for missense single-nucleotide variants (SNVs), which account for the majority of VUS. In Clinvar, one of the most widely used genetic databases, there are about 5% variants with category or clinical significance conflicts (e.g. benign versus pathogenic) among the variants with at least 2 submissions[4]. According to the commonly sequenced genes, the proportions of variants with uncertain significance or conflicting information are even higher (e.g. BRCA1 52% uncertain and 3% conflicting, as of September 2023)[5].

Various technologies that can assess the functional effects of mutations have emerged[6], [7], [8], but only a tiny fraction of the human disease-related genes have been explored. Experimental approaches can be quite expensive and time-consuming when applied to all the remaining VUS, which limits the clinical application of genetic information and the realization of a more efficient discovery tool. Therefore, generating state-of-the-art computational functional predictors for VUS remains crucial.

Evidence suggests that in silico analysis can accelerate the clinical interpretation of VUS[9], [10], [11]. State-of-the-art missense variant functional predictors have been developed by artificial intelligence (AI) [12], [13]. Notably, breakthroughs in protein structure prediction based on deep learning, such as AlphaFold2[14] and RoseTTAFold[15], have improved AI variant predictors by adding information of protein tertiary structures[16]. Here, we review the recent advances in AI prediction for missense variants, particularly protein structure-based methods, and elucidate the challenges and opportunities for this promising direction.

2. Recent developments in the determination of protein tertiary structures

Proteins are polymers of amino acids linked by peptide bonds that fold into specific spatial configurations. The functions of proteins are closely associated with their atoms and amino acid coordinates (tertiary structure), particularly the active domains. Protein structures have been recommended as a strong source of information in the Guidelines for Variant Classification[17]. Researchers have developed several experimental techniques to determine protein structures, such as X-ray crystallography, nuclear magnetic resonance (NMR) spectroscopy, and cryo-electron microscopy (cryo-EM) [18], [19], [20]. While the number of structures stored in the Protein Data Bank (PDB)[21] has steadily increased to 210,000 (as of September 2023), it covers only about 17% of all human protein residues[22].

Since there are a large number of homologous sequences for the proteins that actually exist in nature, the relationship between the amino acid sequence and the structure of a protein follows a certain regularity. Therefore, various computational tools have been developed to predict 3D protein structures. Protein structure prediction methods commonly fall into two categories: theoretical analysis and statistical analysis. While the traditional prediction techniques, such as homology modeling (HM)[23], are relatively mature, recent advances in AI have initiated new concepts to enhance the quality and proteomic coverage of protein structure models.

In 2020, the AlphaFold2[14] project stood out in the challenging 14th Critical Assessment of Protein Structure Prediction (CASP14), followed by RoseTTAFold[15] and ESMFold[24]. On this basis, several multiple sequence alignment (MSA)-free protein structure prediction methods (e.g. OmegaFold[25] and HelixFold-Single[26]), filled the gap in structure prediction using evolutionary information. Compared with the widespread techniques (e.g. HM), these AI-based algorithms exhibit higher availability for several reasons: (i) there is no need for a close homolog solved experimentally; (ii) the whole-protein structure can be obtained; and (iii) the AI algorithms provide unlimited potential for refining the structures[16]. This breakthrough addressed the significant economic and temporal costs in determining protein structures, thus obtaining enough variant protein structures for structural analyses. Moreover, some human proteins possess intrinsically disordered regions that are not conducive to structural analysis. Therefore, the quality of models predicting protein structure varies across different structural domains. AlphaFold is considered to be particularly advantageous in such situations as it provides the predicted local-distance difference test (pLDDT) to reflect the confidence level of each atomic coordinate. It is preferable to use high-confidence regions to develop structure-based variant prediction models. Alternatively, visual inspection algorithms, such as phenix.process_predicted_model and ISOLDE may also be utilized[27].

3. Missense variant prediction using AI

Table 1 presents various state-of-the-art computational methods, many of which can be accessed through a web server. AI-based variant effect predictors based on primary amino acid sequences have emerged in the past few years. There are two main computational approaches. The first approach, supervised training, relies on clinical labels of pathogenic versus benign variants[28], [29], [30], [31]. Given the influencing factors including label bias, label sparsity, label noise, and data leakage, this approach causes inflated prediction accuracy in the specific testing scenarios. The second approach, which involves unsupervised models of evolutionary sequences, has significantly contributed to the advancement of predicting the functional effects of variants[12], [32], [33]. These unsupervised generative models predict variant effects directly from MSA without relying on labels, which grants its theoretical generalizability. However, these sequence-based models have been limited in their ability to address clinical variant interpretation due to the lack of understanding of the protein structures that are more related to function[34].

Table 1.

State-of-the-art computational methods for variant interpretation.

Predictor	Structure accepted	Predicted structure accepted	△△G accepted	Website
DEOGEN2[28]	N	N	N	http://deogen2.mutaframe.com/
PERCH[29]	N	N	N	http://BJFengLab.org/
REVEL[30]	N	N	N	https://sites.google.com/site/revelgenomics/
CADD[31]	N	N	N	https://cadd.gs.washington.edu/
EVE[12]	N	N	N	https://evemodel.org/
EVmutation[32]	N	N	N	http://evmutation.org/
SIFT[33]	N	N	N	http://sift-dna.org/sift4g
AUTO-MUTE[41]	Y	Y	Y	http://binf.gmu.edu/automute/
CUPSAT[42]	Y	Y	Y	http://cupsat.tu-bs.de/
DDGun3D[43]	Y	Y	Y	https://folding.biofold.org/ddgun/
Dynamut2.0[44]	Y	Y	Y	https://biosig.lab.uq.edu.au/dynamut2/
I-Mutant2.0[45]	Y	Y	Y	https://folding.biofold.org/i-mutant/i-mutant2.0.html/
MutPred2[13]	Y	N	N	http://mutpred.mutdb.org/
PMut[46]	Y	N	N	http://mmb2.pcb.ub.es:8080/PMut/
VIPUR[47]	Y	N	N	https://osf.io/bd2h4/
gMVP[48]	Y	N	N	https://github.com/ShenLab/gMVP/
PolyPhen-2[49]	Y	N	N	http://genetics.bwh.harvard.edu/pph2/
SNAP2[50]	Y	N	N	https://rostlab.org/services/snap2web/
AlphScore[36]	Y	Y	N	https://github.com/Ax-Sch/AlphScore/
Missense3D[75]	Y	Y	N	http://missense3d.bc.ic.ac.uk/
SNPMuSiC[76]	Y	Y	N	https://soft.dezyme.com/
SNPs&GO3d[77]	Y	Y	N	https://snps.biofold.org/snps-and-go/snps-and-go-3d.html/
vERnet-B[16]	Y	Y	N	https://ai-lab.bjrz.org.cn/vERnet/
AlphaMissense[52]	Y	Y	N	https://console.cloud.google.com/storage/browser/dm_alphamissense/

Due to the non-negligible contribution of 3D structure to protein function, the local structural properties have been widely used to be combined with sequence features. The combination of structural features showed significant improvement in the performance of variant prediction. Protein structure-based learning models can be classified into feature-based machine learning and graph-based deep learning. Classical machine learning methods rely on features that are manually designed by experts, such as transmembrane helices signal peptides or other motifs[35], [36]. Deep learning methods usually consider structural data as graphs or images and extract task-specific features directly using convolutional neural networks (CNNs)[37], [38] or graph convolutional networks (GCNs)[39], [40]. Structure-based algorithms for predicting the functional effect of one amino acid substitution (Fig. 1) can be divided into two categories of methods based on their utilization of free energy. Energy-based methods utilize experimentally-measured differences in free energy (ΔΔG) between wild-type (WT) and variant structures to train prediction models[41], [42], [43], [44], [45], whereas non-energy-based methods directly use structural features such as hydrophobicity and surface accessibility[13], [46], [47], [48], [49], [50].

Fig. 1.

Diversity of approaches adopted by structure-based variant prediction. The figure illustrates the main sources of information and multiple strategies that were used to develop variant effect predictors.

Building upon the increased structural coverage achieved by incorporating AlphaFold models, our group successfully applied the predicted structure models to identify pathogenic missense SNVs[16]. This work demonstrated the potential of utilizing AlphaFold2-predicted protein tertiary structures for rich feature learning, although they had been considered to fall short in predicting the effect of point mutations[51]. Furthermore, AlphaMissense introduced an additional approach derived from AlphaFold2, enabling the development of a proteome-wide variant effect prediction method by leveraging the structural information[52]. AlphaMissense demonstrated an innovative approach to variant effect prediction by incorporating protein structural information which enabled the inclusion of the two key capabilities of AlphaFold2: the high-precision protein structure model and the ability to learn evolutionary constraints based on Evoformer block.

4. Construction of datasets for machine learning

For supervised AI prediction, benchmark datasets are used as information sources for training and testing models. The supervised variant prediction methods follow two broad strategies according to the type of training labels. The first class of methods directly leverage the variant interpretation on human-curated variation databases, which are generated by experimental assays or clinical evidence. As per the basis of classification, these variation benchmark databases can be further divided into effect-specific databases and disease-specific databases. Variants are classified in effect-specific databases according to the protein's specific function, such as stability[53], [54], [55], solubility[56], and binding free energy[54], [57]. Disease-specific databases represented by Clinvar[5], collect disease-associated phenotypes of variants by clinical data sharing or functional assessments related to pathogenic mechanisms.

The second class of methods avoid using human classification and train with weak labels instead[58], [59]. In this strategy, variants are defined based on their frequencies observed in human or other primate species. These approaches[52], [59], [60] mitigate the impact of biases introduced by human annotation and prevent data leaks between training and testing sets. Specifically, it is supported by the assertion that most of the common variants in primate species are clinically benign in human. Notably, as a result of the incorporation of numerous false labels, these methods still necessitate the use of known labels to assess their actual performance. Table 2 lists the databases widely used in protein effect prediction tasks.

Table 2.

Widely used variation benchmark databases for developing variant effect predictors.

Database	Description	Category	Website
ProTherm[53]	Thermodynamics of protein mutants	Effect-specific	https://web.iitm.ac.in/bioinfo2/prothermdb/
SKEMPI2.0[54]	Protein-protein binding energy, thermodynamics, and kinetics of protein mutations	Effect-specific	https://life.bsc.es/pid/skempi2/
MPTherm[55]	Thermodynamics of membrane protein mutants	Effect-specific	https://www.iitm.ac.in/bioinfo/mptherm/index.php/
Dataset used for PON-SOL[56]	Solubility of amino acid substitution	Effect-specific	http://structure.bmc.lu.se/VariBench/solubility.php/
PROXiMATE[57]	Thermodynamics of protein-protein complex mutations	Effect-specific	http://www.iitm.ac.in/bioinfo/PROXiMATE/
Dataset used for WALTZ-DB[78]	Amyloid forming for variants of hexapeptides	Effect-specific	http://structure.bmc.lu.se/VariBench/amyload1.php/
Nabe[79]	Binding energy of protein-nucleic acid complex	Effect-specific	http://nabe.denglab.org/
Clinvar[5]	Clinical significance of disease-related gene variants	Disease-specific	https://www.ncbi.nlm.nih.gov/clinvar/
UMD[80]	Mutational landscape and significance across 12 major cancer types	Disease-specific	http://www.umd.be/VHL/
IARC TP53[81]	Phenotypes of TP53 mutants in different human tumors	Disease-specific	http://p53.iarc.fr/
DoCM[82]	Validated cancer-causing mutations	Disease-specific	http://www.docm.info/
OMIM[83]	Overviews of genetic phenotypes for disorders	Disease-specific	https://omim.org/
HGMD[84]	Published gene lesions responsible for human inherited disease	Disease-specific	https://www.hgmd.cf.ac.uk/ac/index.php/
BRCA Exchange[85]	A global resource for variants in BRCA1 and BRCA2	Disease-specific	https://brcaexchange.org/
gnomAD[58]	Allele frequencies of variants among human populations	Frequency	https://gnomad.broadinstitute.org/
Great ape genome sequencing project[59]	Genetic diversity among great ape populations	Frequency	http://biologiaevolutiva.org/greatape/

5. Challenges in the development of structure-based variant prediction

Several studies[11], including ours, have observed significant variations in the accuracy of a particular method when being applied to different datasets. The training datasets used in this context are typically derived from experimental or clinical databases. Since these databases are regarded as benchmarks, the potential error labels in the benchmark databases arise the first challenge. This is evident through the inconsistent interpretations of several variants across different databases, indicating a need for caution during data interpretation. Besides the basic approach of deleting ambiguous data, our work has carried out a cross-training solution to eliminate the influence of the remaining unfaithful samples[16]. Another commonly used method is to train with weak labels instead of human-curated classification[52], [59], [60].

Another challenge is the class imbalance of variants in nature, especially between the pathogenic and benign ones. Due to their specific functional roles, some proteins are enriched in benign variants whereas others are enriched in pathogenic variants[61]. However, the skewed datasets can lead to a selection bias toward positive or negative samples. It is therefore essential to incorporate more efficient methods, such as EasyEnsemble[62], SMOTE[63], AdaCost[64], and RUSBoost[65], to address the imbalance by effectively using samples from the majority class. Class imbalance problems can be addressed at the data level, algorithm level, or through hybrid approaches[66]. These methods can be further grouped into the following techniques: re-sampling (under/over-sampling), cost-sensitive learning, and ensemble learning.

Inevitable conflict exists between precision and generalization ability, as highlighted by previous studies[16], [52]. Compared to gene-specific techniques such as vERnet-B, generalizable methods like AlphaMissense do not necessitate the training of a separate recognition model for every protein. However, it may result in imprecision in identifying the effects of minor structural alterations caused by the substitution of a single amino acid. Furthermore, conformational changes caused by a mutation can potentially affect protein function via a variety of mechanisms[67], indicating that the same structural alteration may lead to completely opposite phenotypic consequences for different proteins. Therefore, further investigations on these methods are warranted to achieve a better balance between precision and generalization.

A challenge in implementing the structure-based deep learning is the representation of protein structures. These structures consist of polypeptide chains that can be hierarchically organized into primary, secondary, tertiary, and quaternary structures[68]. Due to such complexity, various protein representations can be used for deep learning, including molecular graph[69] and 3D projection[70] based on the protein’s original 3D shape (Fig. 2). AlphaFold-derived methods can utilize the AlphaFold context[52], which is AlphaFold’s intrinsic understanding of structure, to further learn its relationship with function. Constructing the amino acid networks (AANs)[16] is another mean to provide more detailed structural information, thus enabling the presentation of protein structures for deep learning purposes.

Fig. 2.

Representations of protein structure for deep learning. Molecular graph (a graph whose nodes are atoms and edges are bonds), 3D projection (a 3D array reflecting the shapes measured in three directions), AAN (an undirected weighted network whose nodes are amino acids and edges are their non-covalent interactions), and AlphaFold Context (a structural context generated by AlphaFold, with MSA and Pair representations).

Another limitation of the structure-based approaches is that the existing structure prediction algorithms only provide an optimal static structure, which may not account for conformational changes in proteins resulting from compound binding, complexes, or condensates with various quaternary structures. Although the molecular dynamics (MD) simulation is theoretically appealing, this method remains a huge challenge even for moderately sized proteins[14]. Traditional MD simulations are limited by the computational intractability of algorithms as well as the context dependence of protein stability. Recent studies have demonstrated that more sampling can enhance the prediction of multiple conformational states[71], [72], [73]. However, further evidence is needed to establish the reliability and usability of these methods in revealing the conformational landscape. Nevertheless, the combination of AlphaFold2 with molecular dynamics simulation[74] holds great promise for optimizing structure-based variant predictors.

6. Future direction and concluding remarks

Deep learning for structural and functional prediction is becoming a popular direction, as evidenced by the increasing number of related publications over the last few years[75], [76], [77]. Despite posing challenges[34] and presenting a range of perspectives, previous studies have shown the feasibility of this approach and provided insights for generating even more precise variant effect learning models. The remarkable advances in protein structure prediction not only highlight the potential value of AI in structural biology but also pave the way for future genetic information research with AI. Structural AI prediction promises to tackle the crucial problems in the post-sequencing era and to enrich the toolbox of precision treatment.

In addition, using well-trained models to predict a batch of random mutations can specify the evolutionary tendency of a protein’s function, thereby creating novel variants with the expected function. This approach can expedite drug discovery by reducing the number of wet experiments that validate the function of the selected variants, implying a favorable application of AI-based structure prediction for protein function. Therefore, the ongoing studies are developing novel algorithms specifically for modeling protein structures to enhance the precision of functional prediction, coupled with reliable validation for the newly predicted functional proteins.

Funding

This work was supported by the National High Level Hospital Clinical Research Fund (Grant BJ-2023-077), the National Natural Science Foundation of China (Grant 82372314).

CRediT authorship contribution statement

Fei Xiao: Conceptualization, Writing – review & editing. Lihui Zou: Conceptualization, Writing – review & editing. Chang Li: Writing – original draft, Writing – review & editing. Yixuan Luo: Writing – original draft, Writing – review & editing. Yibo Xie: Writing – original draft, Writing – review & editing. Zaifeng Zhang: Writing – original draft, Methodology, Visualization. Ye Liu: Writing – original draft, Methodology, Visualization.

Declaration of Competing Interest

The authors declare that there are no known competing financial interests.