Critical assessment of methods of protein structure prediction (CASP) — round x

Abstract

This article is an introduction to the special issue of the journal PROTEINS, dedicated to the tenth Critical Assessment of Structure Prediction (CASP) experiment to assess the state of the art in protein structure modeling. The article describes the conduct of the experiment, the categories of prediction included, and outlines the evaluation and assessment procedures. The 10 CASP experiments span almost 20 years of progress in the field of protein structure modeling, and there have been enormous advances in methods and model accuracy in that period. Notable in this round is the first sustained improvement of models with refinement methods, using molecular dynamics. For the first time, we tested the ability of modeling methods to make use of sparse experimental three-dimensional contact information, such as may be obtained from new experimental techniques, with encouraging results. On the other hand, new contact prediction methods, though holding considerable promise, have yet to make an impact in CASP testing. The nature of CASP targets has been changing in recent CASPs, reflecting shifts in experimental structural biology, with more irregular structures, more multi-domain and multi-subunit structures, and less standard versions of known folds. When allowance is made for these factors, we continue to see steady progress in the overall accuracy of models, particularly resulting from improvement of non-template regions.

INTRODUCTION

This article is an introduction to the special issue of the journal Proteins reporting the results of the tenth Critical Assessment of Structure Prediction (CASP) experiment. CASP is an organization that conducts community-wide experiments to measure the state-of-the-art in modeling of protein structure from amino acid sequence. The core principle of CASP is fully blinded testing of structure prediction methods, and that is what CASP has been doing every 2 years, since 1994. The experiment covers an approximately 9-month period. Sequences of proteins for which the structure is about to be solved by X-ray or NMR methods are first solicited from the experimental community. These sequences are distributed to registered members of the modeling community, who submit models before there is any release of the experimental data. Models are then evaluated by a battery of automated methods and assessed by independent assessors.

Experimental structures are currently available for less than 1/1000th of the proteins for which sequence is known, so modeling has a major role to play in providing structural information for a wide range of biological problems.^1,2 During the almost 20 years of the CASP experiments the structure modeling field has changed enormously. In 1994, there were only 229 unique protein folds known (http://www.pdb.org), so that most sequences of interest had no detectable homology to known structures, and could only be modeled by “ab initio” methods. Such modeling was regarded as a “grand challenge” problem in computational biology³ and it was expected that physics methods, together with a better understanding of the process by which proteins fold, would lead to a solution. At that time it was already very clear that, since structures with detectably similar sequences have closely related structures, modeling by homology would be important, but there were relatively few cases where an accurate model could be obtained, and therefore, the field was regarded as largely of academic interest. At present, there are about 87,000 structures in the Protein databank, and these span about 1393 folds, so that a homology model can be produced for in excess of half of all protein domains of known sequence.^4,5 Homology models vary greatly in accuracy depending on a number of factors, and for that reason CASP has encouraged the development of methods that can estimate the likely overall accuracy of a model and accuracy at the individual amino acid level. That, together with testing of modeling methods themselves, has led to wide acceptance of models as legitimate and well-characterized sources of information on structure. Also important has been the emergence of robust on-line⁶ and off-line^7–9 user-friendly modeling software packages, and the provision of databases of models.^10–12 The accuracy of homology models, as monitored by CASP, has improved dramatically, through a combination of improved methods, larger databases of structure and sequence, and feedback from the CASP process. Ab initio modeling methods have also improved substantially, from a very low base in the first CASP experiment. It is now not unusual to see topologically accurate models for small (<100 residues), regular, and single domain non-template proteins.¹³ Very few new structures of such proteins are now appearing, so this capability in itself does not find wide application. However, these methods have become useful in building those parts of homology models that were not easily obtained from a template, a key modeling area which has seen considerable advance in recent CASPs.¹³ Physics and knowledge of the protein folding process have not played a major role in these advances. Refinement of initial models is also an area where more physics-based approaches are expected to contribute. CASP has focused on the issue of refinement and encouraged members of the physics community to become involved, and these efforts bore fruit in CASP10, as outlined later, and reported more in Ref. 14. CASP also monitors progress in several other areas, particularly identification of disordered regions in proteins, and the ability to predict three-dimensional (3D) contacts that can be used as restraints in constructing 3D models. Specifics are outlined below, and reported more fully in other articles in this issue. Particulars of the previous nine CASP experiments can be found in the corresponding Proteins special issues.^15–23

This article outlines the structure and conduct of the CASP10 experiment. It is followed by a paper describing the procedures and model evaluation methods used by the CASP Prediction Center.²⁴ Next is a paper²⁵ describing the CASP10 target proteins, guidelines for splitting these into domain-based evaluation units, and general principles for assigning the relative difficulty of constructing an accurate model in each case. Then there is a paper highlighting some of the most challenging CASP10 targets from the perspective of members of the experimental community who submitted targets.²⁶

As is standard for four CASPs now, targets are divided into two categories of difficulty. One category is for template-based modeling (TBM), where a relationship to one or more experimentally determined structures could be identified, providing at least one modeling template and often more. There is a paper from the assessment team for that class of models.²⁷ The second category is free modeling (FM), where there are either no usefully related structures, or the relationship is so distant that it cannot be detected. As fewer and fewer new folds are discovered experimentally, targets in the FM category have become increasingly difficult to obtain. To address this problem, starting in December 2011, CASP introduced a mechanism by which FM targets are continuously solicited from the experimental community and immediately presented to the prediction community, in a procedure known as CASP ROLL. The CASP10 FM assessment team evaluated models for these targets together with the CASP FM targets from the CASP10 prediction season and there is a paper describing their findings.²⁸

Six other categories of modeling were evaluated. New in this CASP is a “contact-assisted” category. Modeling methods have proven to be instrumental in solving structures based on NMR data in the form of distance restraints or only chemical shift information,²⁹ and new experimental methods, using cross-linking³⁰ and surface labeling,³¹ are also beginning to provide sparse structural information. The idea in the CASP contact-assisted category is to investigate how much experimental information is needed to deliver what level of model accuracy, and to encourage the development of new methods for this purpose. A separate article describes the outcome of the assessment of the 3D models built with the assistance of sparse contact information.³²

As in three recent CASPs, refinement of initial models was also included as a category. Relatively fine scale tuning of models is the end game in modeling³³ and is critical to the production of final structures that rival experiment in accuracy. For this reason, refinement receives special emphasis in CASP, including evaluation of the outcome by an independent assessment team. In this category, selected best models submitted in the TBM category were provided as starting structures, and participants were invited to see if they could improve these. There is an article reporting the work of the assessment team in this category also.¹⁴

In experimental work of any kind, one is almost always obliged to provide estimates of accuracy. In computer modeling of protein structures, in spite of very widely varying accuracy, historically, that was not the case. Emphasis on this aspect of modeling over the last four CASP experiments has led to development and testing of a number of quite effective methods. An article describes the evaluation of these in CASP10.³⁴

CASP continues to evaluate the effectiveness of methods for predicting which parts of a protein do not exhibit a single 3D structure—that is, they are in some sense disordered.³⁵ Although there has been little change in the accuracy of methods for six rounds of CASP, their importance in the real world of modeling continues to increase. An article ³⁶ describes results for CASP10.

The identification of amino acid residues in proteins involved in binding small molecule ligands can provide valuable hints for their functional characterization, as the function of a protein often depends on specific interactions with other molecules. An article reports on the evaluation of binding site prediction methods in CASP10.³⁷

Finally, there is an article³⁸ on the effectiveness of methods for predicting 3D contacts in protein structures. These methods have always been considered potentially important in structure modeling, though performance has not changed much recently. In the last 3 years a number of new methods have been published,³⁹ and the explosion of sequence data has also facilitated the use of deeper alignments, the core input to most methods, leading to strong renewed interest in the possibilities.

The assessment papers are followed by five papers from some of the more successful modeling groups. As in CASP9, contributing modeling groups were asked to concentrate on details of the methods rather than describing the results.

An additional article⁴⁰ describes a method developed by the TBM assessment team to objectively determine which parts of NMR experimental structures are sufficiently well determined that it is reasonable to evaluate the accuracy of models.

The last article in the issue considers the results of this CASP experiment in the context of the previous ones, and discusses progress.¹³ As always, the assessors’ articles are the most important in the issue, and describe the state-of-the-art as they found it in CASP10.

THE CASP10 EXPERIMENT

The structure of the experiment was very similar to that of the earlier ones, described in previous articles in this series.²³ Participants registered for the experiment in two ways: as human-expert teams, where a combination of computational methods and investigator expertise may be used; or as servers, where methods are only computational and fully automated, so that a target sequence is sent directly to a machine. Investigators may register in both categories and limited additional groups may be registered by the same igroups to allow for testing of different methods. The expert groups are allowed a longer time period (typically 3 weeks vs. 72 hours for servers) between the release of a target and submitting a prediction. There are now very few groups where significant human expertise is brought to bear, and the longer period is primarily utilized in two ways—to make use of initial models produced by the rapid server stage, and to perform longer calculations.

Information about “soon to be solved” structures was collected from the experimental community and passed on to the modeling community. As is customary, the main CASP prediction season lasted for 3 months, from May through July. The new ROLL arrangements for template FM targets ran longer, from December 2011 until the end of the CASP prediction season. Continuing the trend of recent CASPs, about 80% of regular CASP targets were obtained from the structural genomics community, primarily the NIH Protein Structure Initiative centers (the PSI, http://www.nigms.nih.gov/Initiatives/PSI). ROLL targets had a higher fraction contributed by the broader structural biology community (11 out of 18). The ROLL experiment is continuing. The PDB now provides an ongoing system for depositors to identify a structure as a CASP target, greatly helping the flow of the process.

Groups were limited to a maximum of five models per target, and were instructed that most emphasis in assessment would be placed on the model they designated as the most accurate (referred to as “model 1”), particularly for TBM. The models were compared with experiment, using numerical evaluation techniques and expert assessment, and a meeting was held to discuss the significance of the results.

MANAGEMENT AND ORGANIZATION

The CASP organization was essentially unchanged from CASP9, and fuller details can be found in Ref. 23. The organizers are the authors of this article. One important change is the addition of Torsten Schwede to the organization team, who joined immediately after the CASP9 meeting. A discussion site (FORCASP) provides a forum for participants (www.FORCASP.org). There is an advisory board composed of senior members of the modeling community who advise the organizers on aspects of the CASP experiments and related activities. A participants’ meeting during each CASP conference allows for more direct interaction, including votes on issues of CASP policy. The Protein Structure Prediction Center is responsible for all data management aspects of the experiment, including the distribution of target information, collection of predictions, generation of numerical evaluation data, developing tools for data analysis, data security, and maintenance of a web site where all data are available. A corner stone of the CASP system is the use of independent assessors to judge the quality of the models received, and interpretation in terms of progress and bottlenecks.

TARGETS AND PARTICIPATION

In the main CASP experiment 114 protein sequences were released as modeling targets, of which 53 were designated “all groups” (human and server) targets, 18 targets were cancelled, leaving 96 where the experimental structures were available for evaluation and assessment. In cases where significant domain movements where observed, or individual domains were classified in different categories (FM, TBM), the targets were divided into separate evaluation units. In all, 131 evaluation units were included. For 28 TBM domains, selected models were released as starting points for the refinement exercise, and for 15 FM or harder TBM domains, sets of contacts were released after the initial models had been collected, to test the extent to which these could guide modeling.³²

The level of participation in the CASP experiment remains high, with 217 registered groups representing a large fraction of the relevant community.

COLLECTION AND VALIDATION OF MODELS

There were a total of 66,297 models deposited in CASP10, of which 45,836 were 3D co-ordinate sets. The remaining submissions are for residue–residue contacts (2514), structural disorder (3136), binding site identification (1817), and estimation of 3D model quality (7679). About 4320 3D structures were refinements of initial models and 1005 were structures contact-assisted models. All predictions were submitted to the Prediction Center in a machine-readable format. Accepted submissions were issued an accession number, serving as the record that a prediction had been made by a particular group on a particular target.

NUMERICAL EVALUATION OF PREDICTIONS

The well evolved standard CASP numerical evaluation methods were again used,^41–44 and two new measures, CAD⁴⁵ and IDDT⁴⁶ were added. For each model, values for all metrics were provided to assessors and subsequently released through the Center web site.

As always, assessors were encouraged to develop their own additional measures to complement the established CASP ones, a process that has led to many new and useful approaches over the course of the experiments. In this experiment, a new metric was introduced by the template-based assessment team, based on their previous work comparing NMR structures. The template FM assessment team built on procedures introduced in previous CASPs to provide a well-integrated and tested package. Collaboration with the UCSF Chimera team⁴⁷ also resulted in a useful graphics tool for rapidly inspecting large numbers of models, a very labor intensive part of the assessment process.

The key principle of CASP has always been that primary responsibility for assessing the significance of the results is placed in the hands of independent assessors. This continues to be a major source of insight and innovation in CASP, as well as ensuring that organizer biases are not imposed on the outcome. In CASP10, the TBM assessment team was led by Gaetano Montelione (Rutgers University, NJ); for Template free modeling, BK Lee (NCI/NIH, BD) and for Refinement and physics-based prediction methods, David Jones (University College London, UK).

MEETINGS, WEB SITE, AND PUBLICATIONS

A planning workshop was held before the start of the CASP10 experiment, attended by the CASP9 and CASP10 assessors and the organizers, with the goal of briefing the new assessors, and deciding on procedures and rules to be followed. A second planning workshop was held about 2 months after the close of the modeling season, at which the assessors presented their results to each other and to the organizers. As always, the identities of participating groups were hidden from the assessors until after those presentations, to avoid ranking bias.

The meeting to discuss the outcome of the experiment was held in Gaeta (Italy) in December 2012. To celebrate 10 successful CASP experiments, the meeting included eight keynote talks from members of the modeling community who have made major contributions both to the field and to CASP. The speakers were Janet Thornton (EBI, Hinxton, UK), David Jones (University College London, UK), Michael Levitt (Stanford), David Baker (University of Washington, Seattle, WA), Joel Sussman (Weizmann Institute, Israel), Nick Grishin (University of Texas, Southwestern Medical Center, Dallas, TX), and Roland Dunbrack (Fox Chase Cancer Center, Philadelphia, PA). In addition to sessions devoted to the outcome of the experiment in each of the modeling categories, the meeting again emphasized discussion of methods, with talks selected by the participants on the basis of the abstracts. In addition to talks by representatives of some of the more successful prediction groups, there were several round table discussions to further probe methods and to discuss directions for future progress. The full program can be found on the Prediction Center web site.

This issue of PROTEINS is the official report of the CASP10 experiment and the outcome of the meeting. All the modeling and assessment papers in this issue have been peer reviewed. The CASP web site (http://predictioncenter.org) provides extensive details of the targets, the predictions, and the numerical analyses.

PROGRESS IN CASP10

The most significant improvement seen in CASP10 was in the Refinement category, where for the first time, one prediction group succeeded in improving the accuracy of all the targets.⁴⁸ Although the overall average improvement was not large, there are impressive examples for particular targets. Encouragingly, this result was achieved with molecular dynamics methods, showing that the more physics-derived approaches are finally making a contribution in modeling. Results in the new category of contact-assisted modeling confirmed that these methods can indeed produce substantially more accurate models with moderate amounts of extra information.³² On the other hand, the new contact prediction methods³⁹ did not result in detectable improvements.³⁸ Two factors may account for that. First, only one group made a serious effort to use these. Second, those approaches are expected to yield most improvement when a deep, well-balanced sequence alignment is available. That was not the case for most targets. We look forward to more representative results in CASP11. Over the last few CASP experiments there has been little obvious overall improvement in model accuracy, for both TBM and template FM. That appeared to be the case in this round using the established scale of target difficulty.⁴⁹ A more thorough analysis of target properties showed that these are increasing in difficulty in ways not well reflected by the standard scale. Particularly for template FM, folds are less regular (e.g., exhibiting a larger radius of gyration) than in earlier CASPs, and more likely to be domains of larger proteins and parts of multi-molecular complexes. For TBM, discoverable templates for CASP10 targets on average provided about 10% less coverage than those in CASP5. The resulting loss of main chain accuracy has been off-set by notably improved methods of modeling regions of the target not covered by the best template.¹³

FUTURE DEVELOPMENTS

The contact-assisted modeling category will be included in the next experiment, enhanced based on experience in CASP10. To better address the specific needs of the ligand binding site prediction category the evaluation procedure has been changed: instead of making binary predictions, the new format allows for predicting continuous probability values, including the specification of ligand type/ligand identity. In order to increase the number of prediction targets, ligand binding site prediction servers are now evaluated continuously using an automated system called Continuous Automated Model Evaluation (CAMEO, http://www.cameo3d.org/), which is based on weekly pre-released sequences from the PDB.

A CASP11 experiment is planned, beginning of spring 2014, and culminating in a meeting in December of that year. The meeting is expected to take place in the United States. Those interested should check the CASP web site for further announcements.