Critical assessment of methods for predicting the 3D structure of proteins and protein complexes

Abstract

Advances in a scientific discipline are often measured by small incremental steps. Here we report on two intertwined disciplines in the protein structure prediction field, the modeling of single chains and complexes, that have over decades emulated this pattern, as monitored by the community-wide blind prediction experiments CASP and CAPRI. However, over the last few years, dramatic advances were observed for the accurate prediction of single protein chains, driven by an upsurge of deep learning methodologies entering the prediction field. We review the main scientific developments that enabled these recent breakthroughs and feature the important role of blind prediction experiments in building up and nurturing the structure prediction field. We discuss how the new wave of AI-based methods is impacting the field of computational and experimental structural biology and highlight areas of future developments that deep learning methods are likely to thrust forward, provided major challenges are overcome.

Introduction

The problem of predicting the native 3D structure of a protein from its amino acid sequence has occupied the center stage in protein modeling research for over 5 decades owing to its inherent scientific interest and to the many potential applications that robust structure prediction algorithms would offer in areas such as the prediction of function from genome sequence, or designing new drugs to treat disease (104). In comparison, although the important functional role of protein-protein interactions and complexes was recognized in the sixties, methods for predicting the structure of complexes became a booming research area only since the turn of the century(136), fueled by the realization of the ubiquitous involvement of protein complexes in nearly all cellular processes.

The last decade has seen major advances in both types of prediction methodologies, due to a variety of factors. Notable has been the application of artificial intelligence (AI) methods, culminating with the recent phenomenal success of the AI-based algorithm AlphaFold2 by DeepMind in predicting the structures of single protein chains to accuracy levels rivaling with experimental methods (53). Important in nurturing and catalyzing these developments have been the blind prediction experiments of CASP (Critical Assessment of Structure predictions and CAPRI (Critical Assessment of PRedicted Interactions), focusing respectively, on the critical assessment of methods for the predicting the structure of proteins and protein complexes.

In this review we outline the progression in the methods developed for these two prediction tasks. We describe how the performance of prediction methods is evaluated by CASP and CAPRI and how progress is assessed. We highlight the role of blind predictions in building up the communities of methods developers and shaping the field. We end by offering our view on the impact the new wave of AI-based methods is having on the field of computational and experimental structural biology, and where the remaining challenges lie.

Prediction of protein 3D structure from sequence

Computational analysis of protein structures has been initiated by Shneior Lifson and his group in the 1960s, who extended the molecular mechanics approach developed for modeling small organic molecules to large molecular systems (42, 43). They introduced the “Consistent Force Field” (CFF) energy function, which led to the development of some of the most important all-atom potentials used today in protein modeling, including CHARMM (11), Amber (133) and ECEPP(50). All three potentials include covalent, non-covalent, and electrostatic energy terms as in the original CFF, with some additional terms, specific to each force field. These classical potentials have served well whenever various intrinsic properties of the protein needed to be investigated in vacuum; however, they were proven inadequate for a thermodynamic description of stable compact protein folds in solution, and unable to discriminate between native proteins and incorrectly folded models (97). The main reason was the failure to account for solvation effects, an important determinant of protein stability. These effects were usually incorporated by using these potentials in molecular dynamics (MD) simulations of the protein immersed in a box of explicit solvent molecules, an exercise, which remained prohibitive for protein structure prediction due to its computational burden, leading to problems of convergence and inadequate conformational sampling.

The next step forward was the addition of implicit solvation terms to the classical potentials. These included surface area-dependent empirical models used in conjunction with atomic solvation parameters (30) and continuum electrostatic terms evaluating the electrostatic contribution to the solvation free energy, using the finite difference Poisson–Boltzmann (PB) method (38) and various approximations to the Generalized Born (GB) treatment (99). Augmented with a surface area-based term to represent the nonpolar contribution to solvation and integrated into the classical potential functions, the resulting force-fields could identify the native states of peptides or proteins, albeit with limited accuracy (45, 147).

Such problems led to interest in extracting effective potentials from experimentally determined protein structures. A frequently used approach to deriving such potentials consists of computing frequencies of structural features (‘structural frequencies’) and converting these frequencies into free energy contributions (123). Following this approach many statistical (or knowledge-based) potentials were proposed (52, 117). Most of these potentials used simplified residue-based representations of the protein, reminiscent of the coarse-grained potentials used decades earlier in protein folding calculations (85). These relatively simple, computationally efficient potentials helped score and rank predicted protein models. When combined with various energy optimization methods they were also able to model the structure of very small proteins from their amino acid sequence, the so-called ab-initio protein modeling approach. But sampling the vast conformational space of average size proteins remained a problem. To address it, data on proteins sequences and known protein structures were increasingly relied upon.

Since evolutionarily related proteins adopt similar 3D structures(22), with the increasing number of experimentally determined protein structures this property gave rise to the method of homology modeling, also known as comparative- or template-based- modeling (29). The atomic-resolution structure of the "target" protein is modeled from its amino acid sequence and an experimental three-dimensional structure of an evolutionary related protein (the template). Aligned regions of the template backbone are simply copied into the target, whereas special prediction methods are used for adding loops in the non-aligned regions (32) and also for placing the side chains of non-conserved residues (32), and the resulting models are refined using molecular mechanics or molecular dynamics methods.

A notable development in conformational sampling, which in some ways bridges template-based and ab-initio methods has been the fragment-based assembly approaches, whereby models are built from short contiguous backbone fragments (typically 3–15 residues in length) taken from proteins of known structure, and assembled into full length models using Monte-Carlo simulated annealing or equivalent techniques (67, 119).

The next major advance in protein modeling has been the effective use of co-evolutionary information, enabled by the growing number of related sequences (39). The underlying hypothesis was that if mutations occurring at two positions in the aligned sequences are correlated, then these positions are likely to form a contact in 3D space (103). To find true evolutionary covariation between residues is difficult because one must minimize the effect of transitive correlations: indirect correlations that are observed, for example, when two residues contact the same third residue but do not actually contact each other. Transitive correlation can be removed by global statistical approaches involving either direct coupling analysis (90), pseudo-likelihood optimization (56), or machine learning (134). The approach has first been used to identify residue pairs that are in contact, and further extended to derive residue distance and dihedral angle distributions, all used as restraints in ab-initio modeling (103, 143). The more recent neural network-based learning methods further extend the use of multiple sequence alignment to end-to-end protein structure prediction, achieving previously unimaginable accuracy for a significant fraction of proteins (53), as will be discussed in the last section of this review.

Prediction of the 3D structure of protein complexes

Efforts to model protein-protein interactions started in the nineteen seventies, driven by the desire to explain aberrant protein-protein interactions caused by a single point mutant in sickle-cell hemogolobin (Hb-S)(84). The first protein docking algorithm, formulated as the task of modeling the atomic structure of a native protein complex from the structures of its components, was developed a few years later. This early incarnation of docking treated the interacting proteins as rigid bodies, used a coarse-grained representation of the protein developed for protein folding calculations (85) and searched for large surface patches with complementary shapes. Shape complementary was evaluated by the interface area formed by the contacting proteins (21), a geometric quantity representing the loss of solvent accessible surface area upon binding, itself related to the hydrophobic contribution to the binding free energy (20).

Ab-initio docking methods

Over the following two decade a variety of docking procedures were proposed (136, 137), including most notably the Fast Fourier Transform (FFT)-based methods (57) that currently dominate the field of protein docking. FFT-based methods enable speedy coarse-grained rigid-body searches capable of detecting shape complementary, as well as the evaluation of different properties of protein interfaces, such as hydrophobicity (127), or electrostatic and van der Waals interactions (9). Following these advances strategies were proposed to speed up high-resolution searches, required for accurately defining the molecular positions and orientations. These include the use of spherical polar Fourier expansion coefficients, shown to significantly accelerate the search for solutions that optimize properties of generated interfaces (112).

A series of rigid docking algorithms with variations on these fundamental principles (19, 35, 60, 112) underpin most of the docking procedures used today. One in particular forms the basis of a well frequented automatic docking server, ClusPro (62), which enables the reduction of the search space from 6 to 5 degrees of freedom by employing a Fourier transform in polar coordinate space, resulting in a 10-fold speed up over classical FFT approaches without compromising accuracy (100). A few alternative sampling algorithms such as enhanced sampling Monte-Carlo procedures (142), and algorithms incorporating heuristic methods based on Particle Swarm Optimization (PSO) (89) also hold their own.

Also worth mentioning, is the so-called data-driven docking methods, which involve the incorporation of distance restraints obtained from biophysical or biochemical data into the modeling protocol, thereby reducing the search space for the location of the native complex. One of the first approaches using these principles, now operating as a publicly available server (24), is the program HADDOCK (28). Similar restraints-based methodologies have been added to other protein docking methods (102, 132, 139). More recently, inter molecular contacts derived from data on residue co-evolution were also used as restraints in docking calculations with however, modest success (108).

Scoring docking poses

To further prioritize the large number of solutions (often in the thousands) produced by the docking calculations, these solutions are re-ranked using more sophisticated scoring functions. An important requirement for such functions has been that they be able to reliably percolate the most native-like binding modes to the top of the list. Aware of the challenge, the development of scoring schemes has been a major focus over the past two decades. These schemes span a wide range and are often combined with model optimization. Use is being made of atom or residue pair potentials, sometimes in combination with classical potential energy terms, but increasingly implementing different flavours of knowledge-based potentials. The latter are adapted from those developed for the structure prediction of single protein chains (46, 48, 145, 146). Among the most effective, are methods combining knowledge-based potentials with the evaluation of interatomic contact areas using Voronoi tessellation(98), methods that enrich knowledge-based potentials with evolutionary relationships (96), and methods augmented with deep learning models (86) or replaced with such models (111). A notable example is the Rosetta all-atom multi-component energy function (2), for its broad use in various molecular modeling applications including the evaluation of docking models. For many scoring schemes the rank of native-like solutions can be bolstered by clustering the top ranking docking poses based on the similarity of their interfaces and using cluster size and stability to perturbation(61) to rank models alone or as part of more complex ranking procedures (62).

Handling protein flexibility

With rigid-body search algorithms making up the core component of most docking procedures, it not surprising that these procedures do poorly when the interacting proteins exhibit moderate to high levels of flexibility (26). Nevertheless, with modifications to some rigid-body docking algorithms for so called ‘soft docking’ - allowing for some atomic clashes to be alleviated subsequently by standard molecular dynamics - this problem can be alleviated to a point. Another strategy coined ‘ensemble docking’ involves generating ensembles of conformers for individual components of a complex by molecular dynamics(121) or normal mode analysis (27), and systematically docking conformers from both ensembles to one another, with however mitigated results (71).

The inability to adequately address flexibility led to the design of algorithms, which introduced protein backbone movements and sidechain repacking of putative interface residues during the sampling process, with some success recorded for small to medium conformational changes upon complex formation (4, 10). However, to this day, it remains debatable as to whether modelling protein flexibility using available methods improves the quality of docked models sufficiently to justify the typical higher computational cost entailed (107). Nevertheless, new methodologies are continuing to address the problems associated with significant conformational change upon complex formation, some of which include a new wave machine learning approaches (44).

Template-based docking

Template-based docking was borne out by the increasing success of the homology modeling techniques for single protein chains, described above. With the growing number of available experimentally determined protein structures, it was soon realized that homology- or template-based modeling may be extended to pairs of homologous complexes, if at least some of their component parts show a degree of sequence similarity (3). This then led to the concept that the 3D structure of a complex can be modelled directly from the experimentally determined structures of other complexes. The accuracy of the method hinges on sensitive sequence searches and alignment to the appropriate complexed proteins (120). Interestingly, bearing in mind the constraints on accuracy described above, enough experimentally determined protein complex templates are available to model most native protein-protein interactions for any organism (70) - a concept underlined by the successful employment of the methodology in a number of recent CAPRI blind trials (74). Rapid searching for homologous complexes is now supported by annotated databases of such predicted relationships (69).

Scoring and ranking models derived from template-based docking conforms to the same principles as for classical docking, but with the potential advantage of having to score and rank fewer models. However, there is an obvious caveat, if the modelled complexes do not comply with the principle of conservation of homologous interfaces, the native complex will not be sampled. This contrasts with the ab-initio methods, where there is always a chance, provided flexibility does not dominate, of at least having a near-native model in the complete list of models generated. Moreover, there are clearly certain categories of interactions that are not conducive to this form of docking, the classic examples being antibody antigen complexes, where evolutionary relationships between the binding partners is not expected to be prevalent, thereby enforcing ab-initio docking methods (40).

Ab-initio and templates-based docking methods are clearly not mutually exclusive, and procedures are actively being developed to employ both in order to model the widest range of interactions possible, and to the highest level of accuracy, should the appropriate levels of sequence homology prevail (140); such pipelines have already been encoded into some automatic docking servers (109, 141). With the recent advances in deep learning approaches (see final section), both for components and full complex modelling, these two principal methodologies are likely to become seamlessly merged.

The blind prediction challenges

CASP

Critical Assessment of protein Structure Prediction (CASP) is a community-wide double-blind experiment for testing and comparing protein structure prediction (94). Every two years sequences of soon-to-be experimentally determined protein structures are collected and passed on to registered predictors. Predictors fall into two categories: teams of participants who usually have a period of three weeks to complete their work; and automatic servers, which must return a model within 72 hours, in principle without human intervention. Predictions are evaluated by independent assessors using well developed criteria. CASP provides research groups with an opportunity to test their protein structure prediction methods and delivers an independent assessment of the state of the art in protein structure modeling to the research community and software users. The results show what progress has been made during the previous two years and expose where future approaches should focus to improve the methodology.

The CASP experiment had a fairly modest start in 1994 with 35 participating research groups (95). Targets were provided in three prediction categories: comparative modeling, fold recognition or threading, and ab initio folding (95). The results of CASP1 demonstrated a sobering failure of prediction methods using physics-based potentials, shocking the protein folding community. The only meaningful predictions have been obtained using comparative modeling, and only for easy targets with closely related known structures. Such negative results made it difficult for protein scientist to continue publishing theoretical papers without participating in CASP. The pre-eminence of template-based approaches was further emphasized by CASP2 in 1996. By that time CASP has become more recognized as a much needed “blind” experiment that had the potential to introduce a new area of reproducibility and openness in protein structure prediction. The number of participating groups grew to 70, and there was some improvement in the predictions of more difficult targets (Figure 1). Improved sequence alignment tools, the use of multiple templates, and fragment assembly approaches further improved results at CASP3 and CASP4, but after that, improvements remained moderate, essentially until CASP13 in 2018. Although contacts maps based on co-evolutionary information were already present in several methods at CASP10 to CASP12, their effective use with deep learning led to a jump in prediction quality only at CASP13 (2018, about 100 participating groups), particularly for difficulty targets (Figure 1). While this improvement was already very significant, CASP14 in 2020 lead to a revolution by AlphaFold2, a neural network based end-to-end prediction method that will be further discussed in this paper. However, predictions by other predictor groups and servers have also become much better (Figure 1).

Figure 1: Backbone accuracy of the best models in each of the 14 CASP rounds.

Individual target points are shown for CASP14. The two targets with the lowest agreement with experiment are colored blue are NMR structures and red point represent the model of a a subunit of a cryo-EM-derived large heteromeric structure. The agreement metric, GDT_TS, is a multi-scale indicator of the closeness of the Cα atoms in a model to those in the corresponding experimental structure, and is reported as a percentage, ranging from 0 to 100. Because of experimental errors and artifacts, models with GDT-TS >90 are considered compatible with experiment in backbone accuracy (65). Target difficulty is based on sequence and structure similarity to other proteins with known experimental structures. (Reproduced from reference (66) with permission).

The CASP predictions are evaluated using a variety of quality measures (63) that are listed on the Protein Structure Prediction Center website (https://predictioncenter.org/). The most important measure is GDT_TS (Global DistanceTest Total Score), shown in Figure 1. Another important measure is the Local Distance Difference Test (lDDT), a superposition-free score that evaluates local distance differences of all atoms in a model, including validation of stereochemical plausibility. However, the ranking of CASP predictions is generally based on GDT_TS, computed over the alpha carbon atoms and reported as a percentage, ranging from 0 to 100, with higher values indicating a closer fit of a model to a given reference structure (see Figure 1 for details).

At CASP14 the targets were assigned to one of four classes of modeling difficulty, based on sequence and structure similarity to already experimentally determined structures: ‘TBM-Easy’ for straightforward template modeling targets, ‘TBM-Hard’ for more difficult homology modeling targets, ‘FM/TBM’ for those with only remote structural homologies and ‘FM’ (Free modeling) for the most difficult set with no detectable homology to known structures (65). However, with the significant improvement in prediction quality, for the ongoing CASP15 the distinction between template-based and template-free modeling is eliminated. As shown in Figure 1, until CASP13 the predictions were becoming substantially less accurate as the level of difficulty increased. However, this changed in 2018 with the introduction of deep learning methods at CASP13 that were able to yield predictions with GDT_TS over 60 for the most difficult targets. This trend was further strengthened at CASP14 (the black straight-line), as prediction quality starts at a GDT_TS of about 95, and rarely goes below 80. Although the outstanding performance at CASP14 is dominated by AlphaFold2, the dashed black line in Figure 1 shows that other groups also made substantial advances.

As shown in Figure 1, at CASP14 automated servers had similar performance to human groups (without the results for Alphafold2). Lastly, we note that the quality of many protein structure prediction servers is continuously evaluated by CAMEO (Continuous Automated Model EvaluatiOn), a fully automated assessment platform, which is a complement to the bi-annual CASP experiment (41, 113).

CAPRI

CAPRI (Critical Assessment of Predicted Interactions) is a community-wide experiment inspired by CASP. It was established in 2001 (51) to offer computational biologists the opportunity to test their algorithms in blind predictions of experimentally determined 3D structures of protein complexes, the ‘targets’, provided to CAPRI prior to publication. Experiments focusing on this prediction task were attempted only twice before, including once in 1996 by CASP (51), attracting limited interest. Since its inception, CAPRI prediction rounds have been managed in collaboration of the PDBe (Protein Databank Europe), at the European Bioinformatics Institute (EBI) ( https://www.ebi.ac.uk/pdbe/complex-pred/capri/ )

Due to the slower rate at which structures of protein complexes are being determined and offered as targets for prediction, CAPRI runs prediction rounds on a rolling basis, as targets become available (51). Like in CASP, participants include automatic servers, which must return models within 72 hours, and human predictors who are given 6-8 weeks to complete their predictions. Initially limited to homo- and hetero- protein-protein complexes, the panel of targets diversified to include protein-peptide interactions, complexes of proteins with RNA, DNA and oligosaccharides(78). With time, target size and complexity also increased, especially with the availability of large multimeric complexes solved to high resolution by cryo-EM. Recognizing the essential role scoring functions play in identifying native-like association modes, CAPRI also offers a scoring challenge upon the completion of each prediction round. In this challenge a larger set of anonymized models predicted by different groups and comprising both correct and incorrect binding modes, is made available to all participants to test new scoring functions independently from docking calculations (78, 82). The data comprising the consolidated ensemble of predicted complexes made available in the CAPRI scoring experiments have been compiled in a freely available benchmark dataset (the ScoreSet(83)), recently extended to include 19000 models of diverse complexes predicted by different methods for all CAPRI targets whose structure has been published.

The way the prediction problem is formulated for a given target has likewise evolved over the years to ensure the best use of the available data on the known structures of single protein chains and complexes. In early CAPRI rounds, predictor groups were offered the coordinates of the unbound structures for the components of the complex to predict. Occasionally the bound conformation of one of the components was provided as input in random orientation with sidechain coordinates stripped away (51). As the repertoire of 3D structures of single protein chains was progressive filled thanks in particular to the structural genomics initiatives (18), participants were invited to predict the structure of the target assembly starting from sequence information alone. This requires the integration of homology-based modeling of individual subunits with docking calculations or relying entirely on template-based modeling when adequate templates for the entire complex are available (see Methods). Identifying adequate templates is not trivial and has been a task members of the CAPRI community, who specialize in ab-initio docking calculations, had to learn to master using available resources such as HHPRED(36), PPI3D(23) or GalaxyWEB(59). The increased reliance on homology and template-based modeling was further catalysed starting in 2014, when CASP included the prediction of protein assemblies in their biannual prediction season in collaboration with CAPRI (80); a collaboration that has been continuing since.

Model quality measures

Objective criteria for independently evaluating the quality of predicted models by comparing them against the target structure, are a key component of blind prediction challenges such as CAPRI and CASP. When dealing with models representing protein complexes comprising two or more interacting partners, the evaluation criteria need to account for both local and global parameters of the molecular assembly. In CAPRI, the standard evaluation protocol involves the evaluation of three parameters for a given pair of interacting subunits (76): Two are based on residue-residue contacts and local backbone similarity respectively, and quantify different aspects of the fit between the model and the target binding interfaces. The third parameter quantifies the relative rigid-body global displacement between the binding partners in the model versus the target. Based on ranges in the values of these 3 parameters, defined by expert evaluation and validated by the community, models are assigned to 4 discrete categories: high-, medium-, and acceptable quality and incorrect (76). For targets representing higher order assemblies with multiple distinct interfaces, submitted models are evaluated by comparing each pair of interacting subunits in the model to each of the relevant pairs of interacting subunits in the target(80). The quality scores of the individual interfaces are then used to derive a global quality score for the full assembly. Several different formulations were tested over time, with the latest being a weighted sum of the quality scores of individual interfaces of the assembly (74).

This interface-centric evaluation reflects the essential role of the binding interface in defining the 3D structure of the complex. Even models that reproduce the structure of the native interface to lower accuracy (acceptable or medium accuracy) provide useful information that can be further exploited, whereas incorrect models that totally miss the binding interface are in general of little utility. The performance of individual groups is therefore ranked based only on the best model of respectively, acceptable- medium- or high- quality each group produces for a given target. This results in a coarse-grained ranking, which has its advantages (79), but may also be deemed cumbersome in comparison to continuous model quality metrics, which can be used to evaluate performance at different graininess levels and are more amenable to statistical analyses. The continuous DockQ metric, formulated as a weighted sum of the CAPRI model quality criteria(7) (see also Figure 2a) is an attractive alternative that CAPRI and other studies(13) have already been using to analyze prediction results. Still missing however, are evaluation criteria that seamlessly integrate quality measures of the interface region with those of the remainder of the protein structure.

Figure 2: Accuracy levels of the best models of protein complexes in CAPRI and CASP-CAPRI prediction rounds, and the relation of the DockQ score to the CAPRI model quality categories.

(a) Scatter plots of DockQ values for models submitted by predictors for individual targets evaluated in ref (78) (vertical axis) as a function of f1 (f1= 2f_nat (1-f_{non_nat})/ (f_nat + (1-f_{non_nat})), where f_nat is the fraction of native contacts recalled in the model, and (1-f_{non_nat}) is the fraction of predicted contacts that are native (79). Individual points are color-coded according to the CAPRI model quality category: incorrect (yellow), acceptable (blue), medium (green), high(red).

(b) DockQ values for the best models as a function of target difficulty. Individual colour coded plots refer to best models evaluated in individual CAPRI assessment periods and CASP-CAPRI prediction rounds between 2009-2020, as indicated in the legend. Examples of targets of different difficulty levels are shown together with their PDB-RCSB codes. Arrows indicate the rounds in which they were offered; numbers shown in parentheses following the PDB codes refer to the models of individual interfaces of the targets in question. Target difficulty is based on sequence and structure similarity to other proteins with known experimental structures.

Evaluating progress

CAPRI has been evaluating progress in intervals of about 3 years, with each evaluation performed on results for 10 – 30 targets achieved during the prediction rounds of the intervening period. In addition, the CAPRI team, which has been evaluating CAPRI prediction results since its inception, independently evaluated the results obtained by participants in the assembly prediction challenges of the CASP11-CASP14 prediction seasons. All the evaluations were performed on results achieved by human predictor groups (~40 on average), by automatic servers (increasing from about 3 to as many as 12 over the years), and by participants in the scoring challenges (15-20 groups on average). Detailed evaluations of these results and assessments of the progress achieved by the community have been reported in the associated publications, amply cited in this review. Here we provide a bird’s eye overview of the main trends.

Figure 2b plots the quality of predicted binding modes measured by the average DockQ score across human predictor groups as a function of the level of modeling difficulty of the corresponding targets. These binding modes were those evaluated in the successive dated periods since 2009, setting the chronological order of the results. The plots clearly illustrate the substantial variability in target difficulty levels during individual evaluation periods that persists over time, highlighting the challenge of evaluating differences in performance across time periods and challenges. Nonetheless, one observes that model quality improves with time for easy and medium difficulty targets but remains low for difficult targets. Examples of targets in different modeling categories and typical characteristics of these targets detailed in Figure 2b, clearly indicate that the modeling challenge differs substantially depending on the system at hand. Some large multi-component assemblies solved to high resolution by cryo-EM (not shown) are particularly challenging to model when they combine several characteristics of difficult to model complexes.

An important contribution of CAPRI community has been the development of automatic servers, the performance of which has steadily improved and diversified to the point of often rivaling with those of human predictors(75). Several of the best performing servers such as ClusPro(62), GalaxyPPDock (73), MDockPP (47) integrate docking procedures with template-based modeling and offer a panoply of handy tools for various modelling tasks, therewith gaining popularity with the wider scientific community.

Another area where CAPRI helped breaking new grounds is the prediction of protein-peptide complexes. This is an important category of complexes for which interest is rapidly growing given the important role recognition of short peptide motifs by protein domains plays in many regulatory processes (128). Recent methods, also implemented in several automatic servers, rose to the task of mastering the problem of modeling this challenging category of transient complexes, often to medium and high accuracy, despite their small binding interfaces, and the significant degree of flexibility of the bound peptide ligands(78).

For the other modeling problems CAPRI occasionally offers, such as the prediction of protein-nucleic acid complexes (81) of interface sidechain conformations and positions interfacial water molecules (77), or for protein oligosaccharide complexes, encouraging results were obtained (78). But the number of targets for these complexes was too low to draw any conclusions.

Considering the state of protein science before these blind prediction experiments, it is difficult to imagine that the current level of prediction technology could have been reached without CASP leading the way. With CAPRI following suit a few years later, both experiments created higher level of transparency not only in protein structure prediction but in computational biology in general, requiring the source code for most publications. They also built competitive and yet collaborative communities, promoting the increased exchange of ideas, and thus speeding up method development.

The breakthrough of DL-based prediction method: The current state of play

The last few years have witnessed a breakthrough in modelling the 3D structure of proteins. This breakthrough can be attributed to two primary factors. One is the extraordinary growth in protein sequence databases (126) coupled with a less prolific, yet notable, growth in the database of experimentally determined structures (131), both freely available in public depositories. The second is the progressive introduction of cutting-edge methods in deep learning to a maturing protein modelling field (5, 122, 124). A key role was also played by the community-wide initiatives that enabled the critical evaluation of the recent breakthrough methods for predicting the structure of single protein chains, recorded in the CASP13 and CASP14 prediction seasons(16). Without these three components the extraordinary achievement of the company Google DeepMind (118) would not have been possible. In the following we examine the impact of this remarkable achievement on charting the structural landscape of native proteins and their complexes using computations and experiments.

Predicting the structure of single chains

For the reasons invoked above, deep learning, a subfield of machine learning that utilises multi-layered artificial neural networks to extract patterns within large datasets, without the need to explicitly define their features (72, 114), was uniquely primed to make substantial contributions to protein structure prediction field. Taking advantage of this, the DeepMind scientists designed and employed their prediction engine AlphaFold2 (53) in the CASP 2020 season (54). This engine was trained on approximately 170,000 experimentally determined protein structures in the PDB (8) and massive data on multiply aligned protein sequences of related proteins, many of unknown structure (126), to produce models that rival in accuracy with experimentally determined protein structures, surpassing the results of their first program, AlphFold1 (116) shown to performed well 2 years earlier (115). The power of AlphaFold2 lies with its novel multi-component architecture that jointly embeds features from multiple sequence alignments (MSAs) and a residue pair representation, encoding spatial relationships between residues, and integrates graph-based components with attention learning (58). The pipeline also involves iterative refinements of predicted residue interactions based upon their predicted interactions to other residues, enabling it to encapsulate structural features to a higher level of accuracy in a fully differentiable end-to-end deep learning method(53).

To enable the community benefit from their monumental advance, DeepMind made their source code for the trained model of AlphaFold2 freely available to anyone wishing to make new predictions [https://github. com/deepmind/alphafold]. Following suit, the group of David Baker released their deep learning protocols for protein prediction and design, RoseTTAfold an algorithm exploring similar ideas to those of AlphaFold (6). In a further move to accelerate scientific research, DeepMind partnered with the European Bioinformatics Institute (EBI), to create AlphaFold-DB(130) providing access to predicted structures of single protein chains for the human proteome and other key organisms, as well as to the majority of the manually curated Uniport entries (SwissProt), with plans to further extend the coverage to over 100 million catalogued proteins, on 2022.

This new treasure trove of structural data and the associated software tools has had a watershed effect on the field of computational and experimental structural biology, generating a flurry of studies (16). Examples include the optimization of multiple sequence alignments fed into AlphaFold2 (12), and a community-wide study evaluating various aspects of the structural information that AlphaFold produced and the applications it enables (1). Evaluating these applications, has been greatly aided by two confidence measures Alphafold2 assigns to its predicted structures: a per-residue measure of confidence assigned to the local backbone structure, and another measuring the confidence associated with residue-pairwise distances (53). The first is usually high for structures domains, but low for linker regions, regions that may be flexible, intrinsically disordered or structured only in the context of a larger complex. The second is useful for assessing more global features, such as domain packing.

Results obtained from these first analyses suggest that AlphaFold2 can be used to substantially extend the structural information for model proteomes beyond what is enabled by homology modeling, provided its confidence metrics are critically interpreted. For example, while the atomic coordinates of regions modelled with low confidence may not be trustworthy, they can nevertheless be used to predict disordered regions more accurately than state-of-the-art methods (1) . Similar analyses, currently underway to characterize the structures of AlphaFold-DB whose number is orders of magnitude larger, should shed further light on the information that may be safely extracted from these predicted structures. Clearly missing however, is information on the dynamic properties of proteins, many of which adopt multiple conformational states that are essential for their function (i.e binding other proteins, nucleic acids, small molecule ligands, or switching between functionally active and inactive states) (93, 138). This is currently a serious limitation of deep learning approaches such as AlphaFold2 as recently discussed (33). Tackling this limitation is an important goal, which is receiving increased attention, as further noted below.

Notwithstanding these limitations, ready access to the predicted protein structures in AlpfaFold-DB and to the freely available code and resources such as ColabFold(88) for predicting new structures with AlphaFold2 or RoseTTAFold efficiently with more modest computational resources, is having a resounding impact on experimental determination of protein structures. Rather than putting experimental structural biology out of business it is offering new opportunities like never before. Combining these opportunities with the recent spectacular advances in cryo-EM techniques, is propelling the field to new levels. In several instances, hard to solve X-ray or cryo-EM structures have been elucidated by using AlphaFold models in molecular replacement protocols, (64, 87). AlphaFold and RoseTTAfold models have been used successfully to fit residual electron density of cryo-EM maps, most notably in a recent assembly of the human nuclear pore complex (91). This is clearly an area that should soon see major advances from closer integration of deep learning-based and other structure modeling approaches with emerging DL-based and experimental cryo-EM techniques (125, 144).

Prediction of protein complexes and assemblies

An obvious next frontier for DL-based protein structure prediction methods is the accurate prediction of complexes and larger protein assemblies, as witnessed by a flurry of recent studies reporting forays towards this goal. Several benchmarking studies suggest that extensions of DL-based methods to the prediction of protein complexes will provide a major advance over traditional docking methods. In these studies AlphaFold2, was ‘tricked’ into successfully modeling the structure of a set of protein complexes of known stoichiometry, albeit not consistently to high accuracy, by feeding it the concatenated sequences of the interacting component proteins. (13). Better performance, was reported for AlphaFold-Multimer the inference engine of AlphaFold, directly trained on protein complexes from the PDB (31). For the same benchmark dataset of heteromeric interfaces AlphaFold-Multimer produced acceptable prediction (DockQ ≥ 0.23 ) for ~67% of the interfaces, but high accuracy predictions(DockQ ≥0.80) for only 23%, an improvement of 25% and 11% respectively, over the modified alphafold2 version. More modest improvement was achieved for homomeric interfaces generally associated with higher binding affinity, for which larger fractions could be predicted to acceptable (69%) and high accuracy (34%), respectively. At the same time ways have been proposed to integrate AlphaFold2 predictions of complexes with classical docking calculations and using the predicted complexes as templates for AlphaFold2 to significantly improve the performance of either method used independently(37).

Furthermore, several studies have suggested that AlphaFold and RoseTTafold, can be used to extend the structural coverage of model interactomes beyond what is enabled by homology modeling (of complexes, or of single chains followed by docking (92)). For example, for the human interactome AlphaFold predicted ~1,400 high confidence models of complexes displaying no homology to a known structure (15). Both prediction methods were used of identify interacting proteins and model their complexes in Baker’s yeast (49) and the human mitochondrion (105), deriving in each case new structural information for functionally important complexes.

While these early results are very promising, they also indicate that significant room remains for improvement. Further optimizing these methods to tackle complexes spanning a wide range of binding partners, binding affinities, and functional states, has the potential to lead to significant breakthroughs for these prediction problems. However, fulfilling this potential will not be effortless. The structural coverage of the protein complexes that form in living cells – the body of data AI methods need to ‘learn’ from – is orders of magnitude smaller than the current structural coverage of single protein chains. Furthermore, the formation of many of the more transient complexes featuring lower binding affinities, such as those associated with signal transaction processes, is highly context dependent. Currently however, the ability of experimental structural biology to adequately sample the physiologically relevant contexts of complexes is limited. Third, modeling the dynamic properties of the component proteins, which govern the conformational changes associated with binding, will be an important bottleneck to overcome.

What next?

Although this new wave of applied deep learning methods in the protein structure prediction field have rocked the biological sciences, with numerous applications in view (55), much work will be needed to mitigate important current limitations. Top of the list is the problem of accounting for dynamic flexibility, within single chains, the association process and within the complex (106). This is important for understanding and modelling the functional states of proteins, including those of intrinsically disordered proteins of which there is a natural abundance (68). Next, and related to the first, DL-based methods are still incapable to interpret the effects of single point mutations; backbone movements are simply not replicated when one amino acid is substituted for another, as benchmarked in several recent studies (14, 101).

Here too deep learning methodologies are offering a way forward by first ‘understanding’ the conformational states a protein samples (f.e. in known structures) and the likely transition paths between them (110). This ‘understanding’ is then used to further extend the sampled conformational space, by generating experimentally unobserved but native-like protein conformations, as recently described with an Autoencoder method (25). These descriptions of multiple conformations of a given protein will have to be integrated with the data on multiple sequence alignments to model structures corresponding to specific functional states. Crucial for training and testing this type of methods will be the development of benchmark datasets of physiologically pertinent structures of single chain proteins and complexes that incorporate information on the sampled conformational landscape.

Addressing these challenges will impact all areas of protein structure predictions including that of protein assemblies. Here progress will also depend on the ability of deep learning algorithms to restrain the sampling of the vast number of potential binding modes closer to the basin of native-like solutions, even in cases where co-evolution signals derived from the multiple sequence alignments are weak (such as for complexes with antibodies, or host pathogen interactions): a problem that AlphaFold seems to struggles with (37). Improving the ability to recognize native-like binding modes – the model ranking problem will likewise be important. Promising results towards the latter goal have recently been reported by standalone deep learning models implementing convolutional neural networks (CNNs) and other methods (17, 111, 135). Ultimately ranking models of protein complexes needs to display some correlation with binding affinities, a goal that scoring methods have pursued with modest success(34, 129).

As end-to-end machine learning methodologies are improved and mastered by the wider structural biology community, it will become routine to model a significant fraction of proteins and the complexes they form just from their amino acid sequences, ultimately negating the need for intermediate steps, such as searching for and utilising closest structural templates. One likewise expects these new methodologies to be extended to modelling nucleic acids, more particularly RNA but also DNA, and the complexes they form with proteins within the cell. Here too a major challenge will be to collate enough experimental data to train and validate machine learning methods. The way forward would be a closer integration of computational and experimental approaches. This would involve combining emerging methods for extracting information on structural heterogeneity in macromolecular complexes from the cryo-EM data obtained from endogenous material with AI-based structure prediction algorithms and molecular simulation techniques. All these are important developing areas where blind prediction initiatives will continue to play a major role.