Static benchmarking of membrane helix predictions

Abstract

Prediction of trans-membrane helices continues to be a difficult task with a few prediction methods clearly taking the lead; none of these is clearly best on all accounts. Recently, we have carefully set up protocols for benchmarking the most relevant aspects of prediction accuracy and have applied it to >30 prediction methods. Here, we present the extension of that analysis to the level of an automatic web server evaluating new methods (http://cubic.bioc.columbia.edu/services/tmh_benchmark/). The most important achievements of the tool are: (i) any new method is compared to the battery of well-established tools; (ii) the battery of measures explored allows spotting strengths in methods that may not be ‘best’ overall. In particular, we report per-residue and per-segment scores for accuracy and the error-rates for confusing membrane helices with globular proteins or signal peptides. An additional feature is that developers can directly investigate any hydrophobicity scale for its potential in predicting membrane helices.

INTRODUCTION

Membrane spanning proteins are vital for cells to function (1,2). However, it is very difficult to experimentally determine high-resolution three-dimensional (3D) structures for these proteins: <50 structures are currently deposited in the Protein Data Bank (PDB) (3,4). C-terminal fusions with indicator proteins (5,6) and from antibody-binding studies (7,8) reveal the location of the helices and the orientation with respect to the protein termini. We refer to these data—slightly incorrectly—as ‘low-resolution structures’. Möller, Apweiler and colleagues at the EBI (9) have carefully hand-selected the results from low-resolution experiments for ∼500 proteins. We have taken the high-resolution set from PDB, added the low-resolution set from the EBI and have filtered the noise from redundancy of very similar proteins by creating the largest possible sub-set that is chosen such that we cannot infer structural similarity between any pair of proteins in that set from sequence alone [sequence-unique subset (10,11)]. Our previous work gives a comprehensive evaluation of transmembrane helix prediction methods based on these sets (10,12). Our automatic benchmarking server accomplishes this by using several different evaluation criteria for evaluating accuracy of the prediction method against proteins of known high- and low-resolution structure. False positives are estimated by applying the method to signal peptides and proteins without trans-membrane helices.

METHODS

Input

The server accepts two types of input from users. Firstly, simple scales reflecting the propensity of residues to form membrane helices, e.g. hydrophobicity scales. Such scales for each amino acid are either uploaded in text format or entered into a form. Secondly, results from novel prediction methods: developers can benefit from the benchmark server by following these steps: (i) download the data sets from our web site (∼2200 proteins, some of which contain membrane helices, some do not); (ii) run your method on all proteins; (iii) upload the predictions to our server in either of two commonly accepted formats. The upload is checked for possible problems that are immediately communicated to the developer. For example, if the predictions contain three-state—helix/non-helix/possible-helix—predictions rather than two state—helix/non-helix—predictions, the user is given a choice to convert all possible-helix residues to non-helix, or convert all possible-helix residues to helix.

Algorithm

When we test hydrophobicity scales, we simply apply the Wimley-White algorithm to turn such scales into predictions of membrane helices (25). New prediction methods (or predictions) are evaluated directly from the data uploaded by the users. A detailed description of the particular scores and schemes explored to measure performance is beyond the scope of this manuscript; they are available in our original publications or on our web site (10,12).

Output

Submissions are tracked through identifiers (IDs) that are shown on all result pages. After the request is queued, the user can either refresh the results page to check the status of the request or follow the link that is emailed to the provided email address when the request is completed. When the results are ready (∼5 min), the user is presented with several tables showing how well the tested method/scale performs in comparison to established methods (Table 1). Results are given separately for (i) high- and (ii) low-resolution membrane proteins, as well as for the discrimination against (iii) globular proteins and against (iv) signal peptides (Fig. 1). In all four resulting tables, several columns show different measures for prediction accuracy and discrimination. Clicking on the column headers will resort the given table by that metric. Clicking on the question mark (?) in the column header names will give a description of the metric. Clicking on the other prediction servers in the row header will give the full name of the server as well as the citation for the source as well as a web link to the server if available. Although the primary format used is the interactive web document described here, the results can also be obtained in non-interactive format. If desired, the results will be emailed in text format along with the link to the interactive results. Additionally, a permanent, non-interactive web document can be generated on the server by clicking a link on the interactive web page. A link to the document is then provided which the user can use to reference the server's results.

Table 1. A list of all advanced methods tested and selected simple hydrophobicity scale methods.

Method (Reference)	Note
DSSP (13)	Program assigning secondary structure
HMMTOP (14)	Hidden Markov model predicting transmembrane helices
DAS (15)	Optimises use of hydrophobicity plots
SOSUI (16)	Uses a combination of hydrophobicity and amphiphilicity preferences to predict membrane helices
TMHMM (17)	Embeds a number of statistical preferences and rules into a hidden Markov model
PRED-TMR (18)	Uses a standard hydrophobicity analysis with emphasis on detecting the beginnings and ends of helices
PHDhtm (19,20)	Combines a neural network using evolutionary information with a dynamic programming optimisation
TopPred2 (21)	Averages the GES-scale of hydrophobicity
Ben-Tal (22)	Hydrophobicity scale representing free energy of transfer of an amino acid from water into centre of hydrocarbon region of lipid bilayer
Kyte-Doolittle (23)	Hydropathy index
Wolfenden (24)	Hydration potential
Wimley-White (25)	Hydrophobicity scale

Figure 1.

Sample for server output. Example of one of the four output tables from the server showing where the tested method falls in the ranking of existing methods. Table shown is for accuracy of predicting helices as tested against known high resolution structures. The method named YOU is highlighted in the output and shows the result for the tested method. Hyperlinks re-sort by the given column and also lead to column (scoring metric) and row (prediction method) descriptions.

DISCUSSION

Standard point of reference

The primary goal of this server is to provide users, developers and referees with a standard benchmark evaluation for helical trans-membrane prediction methods in a format that is publicly available and as convenient as possible. The tool may help all not to over-estimate performance and/or to spot strengths and weaknesses of particular methods. The battery of measures for performance that we use encompasses almost all the scores that are found applied in the literature. With the web server, any new algorithm can be tested instantly and seamlessly.

Downside of static benchmark: over-fit to do well on this set only

We might argue that a possible problem with such an easily available and comprehensive method is that someone with enough time on his hands could write a program searching the space of all possible hydrophobicity-like scales in order to optimise the performance on our sets. More generally, developers may over-fit their models. In fact, to some extent, this is a principle problem of any standard data set accepted in the community. However, we challenge that if the scale/method really does consistently better than all methods in respect to all scores, it may indeed capture important aspects of helical membrane proteins. Perhaps more probable is the possibility that one may accidentally over-fit to the benchmark by testing against the benchmark several times during development. To that end, developers can at least reduce the risk of fooling themselves by first testing their final or nearly final method on their own data sets and by then investigating to what extent their results are confirmed by ours.

Ultimate solution: go dynamic

Nevertheless, there is only one way to completely solve the problem, namely test on proteins that could not have been used to develop methods since their experimental structures arrived after the method. This is the concept that we explore through our EVA server evaluating the performance of structure prediction for globular proteins (26–28). However, for globular proteins 10–50 new structures appear in PDB every week. Although this will not become reality for membrane proteins in the foreseeable future, we are currently investigating ways of embedding some dynamic system for the evaluation of membrane predictions into EVA.