Benchmarking B cell epitope prediction: Underperformance of existing methods

Abstract

Sequence profiling is used routinely to predict the location of B-cell epitopes. In the postgenomic era, the need for reliable epitope prediction is clear. We assessed 484 amino acid propensity scales in combination with ranges of plotting parameters to examine exhaustively the correlation of peaks and epitope location within 50 proteins mapped for polyclonal responses. After examining more than 10⁶ combinations, we found that even the best set of scales and parameters performed only marginally better than random. Our results confirm the null hypothesis: Single-scale amino acid propensity profiles cannot be used to predict epitope location reliably. The implication for studies using such methods is obvious.

Historically, vaccines have been attenuated whole pathogens such as BCG or Sabin’s Polio vaccine. Safety concerns have led to other strategies for vaccine development that focus on antigen and epitope vaccines. The latter use peptides able to elicit antibodies that cross-react with a parent protein (Langeveld et al. 1994). Peptides recognized by protein-specific antibodies have diagnostic applications (Orbach and Shoenfeld 2003). B-cell epitopes are classified as either linear or discontinuous epitopes. Linear epitopes comprise a single continuous stretch of amino acids within a protein sequence, while an epitope whose residues are distantly separated in the sequence and are brought into physical proximity by protein folding is called a discontinuous epitope. Although most epitopes are thought to be discontinuous (Barlow et al. 1986), experimental epitope detection has focused on linear epitopes. The accurate in silico prediction of linear epitopes from protein sequences is highly desirable and is thought to be achievable by using amino acid scales to create sequence profiles (Castelletti et al. 2004; Hua et al. 2004). Though the predictive accuracy of such methods has been examined previously (Hopp 1986; Van Regenmortel and Daney de Marcillac 1988; Pellequer et al. 1994), the extensive and objective analysis we present here is the most comprehensive to date. It focuses on a thorough dissection of the underlying hypothesis: Peaks in single amino acid–scale propensity profiles are significantly associated with known linear epitope locations. We exhaustively compared known epitope locations with propensity profile peaks generated using 484 amino acid scales combined with ranges of profile parameters, and we could find no evidence to support this hypothesis.

Materials and methods

Four hundred eighty-four amino acid scales were taken from the AAindex database (http://www.genome.ad.jp/aaindex) (Kawashima and Kanehisa 2000). Each scale assigns a value to each of the 20 normal amino acids. Scale values were normalized. We tested a set of 50 epitope-mapped proteins. Linear epitopes for each protein were extracted from the Antijen database (formally JenPep; http://www.jenner.ac.uk/AntiJen) (McSparron et al. 2003). Each experiment used polyclonal antibodies raised against the whole protein and resolved epitopes to one to three amino acids. We then verified the exact identity of published and test-set sequences. Mapping information from exclusively autoimmune studies was not included. Predicted and determined epitope and nonepitope residues were compared for all test-set protein residues using counts of true positive (TP), true negative (TN), false positive (FP), and false negative (FN). Prediction accuracy was characterized using four criteria (Baldi et al. 2000):

1. Specificity: 100 × TN/(FP+TN)

2. Sensitivity: 100 × TP/(FN+TP)

3. Matthews correlation coefficient (MCC): the overall accuracy of the prediction with a value between 1 and -1, where 0 represents random correlation (Baldi et al. 2000)

4. Mutual information coefficient (IC): a value between 0 (random correlation) and 1 (either complete agreement or disagreement between the observed and predicted) (Baldi et al. 2000).

Three scale implementation methods were used. For Method 1, each scale was implemented using the “sliding-window” method: a running average of amino acid properties for a defined number of residues along a protein sequence. The mean of the resulting profile was used to determine a cutoff value, enabling identification of predicted epitope residues. The 484 scales were initially tested using a reduced range of window sizes and cutoff values. Negative correlation was examined by inverting each scale. The 50 most accurate scales were then examined more fully using an increased range of variables including six other equations for determining the value for each window.

Method 2 was implemented as for Method 1, except that profiles resulting from different window sizes were examined collectively. Predicted epitopes were identified by selecting the highest scoring nonoverlapping windows after ranking individual window values. The top 50 scales were then examined using a wider range of variables. For Method 3, profile filtering was implemented in order to assess correlation between major profile peaks and epitope locations. Two filtering algorithms were applied independently to individual scales; both were based on an iterative reaveraging of sequence profile values. Major profile peaks were identified as a trend over 7 profile points. Scales were implemented in accordance with Method 1. The 50 most accurate scales identified in the initial round were then examined under a greater range of variables.

Results and Discussion

The literature indicates that amino acid propensity profiling is used routinely to predict or retrospectively identify epitope locations. Several papers have sought to quantify the relationship between the location of peaks and epitopes (Hopp 1986; Van Regenmortel and Daney de Marcillac 1988; Pellequer et al. 1994). The limited number of mapped proteins and the method of epitope identification used may have introduced bias into these results. Moreover, the scoring methods used were not sufficient to characterize fully their predictive performance. Our test set is significantly larger: We used 50 epitope-mapped proteins, as defined by polyclonal antibodies, which were mapped to a three–amino acid resolution. This represents the best impartial nonredundant test set currently available. Each propensity scale was initially examined using 17,100 combinations of algorithm parameters, with 228,900 combinations used subsequently for the 150 most accurate scales. The prediction accuracy calculations used here indicate clearly the performance of each scale.

MCC and IC scores suggest that for even the most accurate predictions the correlation between predicted and determined epitope residues is only marginally greater than random, as illustrated in Figure 1 ▶. This suggests that there is no significant correlation between the sequence profiles generated and the location of known linear epitopes. The hypothesis that sequence profiles generated with a single scale can be used to predict effectively linear epitopes from the primary sequence of proteins is thus not supported by the evidence. Given the complexity of antibody–antigen interaction and the relative simplicity of sequence profiling methods, this finding is not unsurprising, and the implications of our study are clear.

Figure 1.

Specificity vs. sensitivity scores of the 150 optimized scales from the three amino acid sequence profiling methods. The broken line indicates random prediction.

Though anecdotal evidence suggests that profiles can accurately predict linear epitopes in some instances (Castelletti et al. 2004; Hua et al. 2004), our results support the null hypothesis: Amino acid propensity profiles cannot be used effectively to predict linear epitopes. It is possible, although unlikely, that as-yet-undiscovered scales may perform adequately. Given the weak predictive performance of the profiling method demonstrated here, the development of more sophisticated approaches, perhaps using artificial intelligence techniques, is required to address this need and should be an obtainable goal.

Electronic supplemental material

The supplemental material contains a detailed description of the methods with additional references. Figure S1 demonstrates a sequence profile, and Tables S1 and S2 detail the five most accurate scales for each of the three methods.

Article and publication are at http://www.proteinscience.org/cgi/doi/10.1110/ps.041059505.