Steps: A Grid Search Methodology for Optimized Peptide Identification Filtering of MS/MS Database Search Results

Abstract

For bottom-up proteomics there are a wide variety of database searching algorithms in use for matching peptide sequences to tandem MS spectra. Likewise, there are numerous strategies being employed to produce a confident list of peptide identifications from the different search algorithm outputs. Here we introduce a grid search approach for determining optimal database filtering criteria in shotgun proteomics data analyses that is easily adaptable to any search. Systematic Trial and Error Parameter Selection – referred to as STEPS – utilizes user-defined parameter ranges to test a wide array of parameter combinations to arrive at an optimal “parameter set” for data filtering, thus maximizing confident identifications. The benefits of this approach in terms of numbers of true positive identifications are demonstrated using datasets derived from immunoaffinity-depleted blood serum and a bacterial cell lysate, two common proteomics sample types.

Liquid chromatography tandem mass spectrometry (LC-MS/MS) shotgun proteomics has become a widely applied technology for characterization of complex protein samples [1-3]. A critical step in the bottom-up proteomics data analysis pipeline is matching MS/MS spectra to theoretical fragmentation patterns of peptides derived from a protein sequence database. A number of search algorithms, such as SEQUEST [4], X!Tandem [5], and Mascot [6] are available for this purpose. Output from these algorithms is a list of peptide-to-spectrum matches (PSMs) along with metrics that reflect the quality of the match. Although robust, the process is prone to high rates of false positive and false negative peptide identifications. The challenge is to remove the false positives, while retaining the maximum number of true positives [7-11].

All of the database search engines provide PSM scoring metrics that aid in differentiating between true and false matches. Additional information such as mass measurement accuracy, charge state of the ion, number of tryptic termini, and search engine independent scoring algorithms, e.g., MS-Generating Function (MS-GF) [12] can be used to further refine research results. Mass measurement accuracy in particular is helpful for filtering high mass resolution data, and even more so when systematic error correction is employed [13-14]. Common approaches for increasing confident peptide identifications include combining scoring metrics, and partitioning PSMs based on charge state and number of tryptic termini prior to the generation of filtering criteria [10, 15]. The most commonly accepted metric for validating filtering criteria utilizes a decoy database search strategy to estimate the false discovery rate (FDR) [7, 16].

We developed STEPS (Systematic Trial and Error for Parameter Selection) as a means of optimizing filtering criteria to improve confidence in peptide identifications. The STEPS methodology (Figure 1) utilizes a freely available automated processing engine (APE; available at http://omics.pnl.gov/software/APE.php) that allows a user to leverage all information available to them for a particular MS/MS dataset to generate optimal filtering criteria for that dataset. The user specifies the parameter(s) to be used, the range over which to search them, and an increment. The integrated ranger tool uses this information to create an iteration table that consists of all possible combinations of the user-defined criteria. This table is read by APE, which generates the appropriate SQL statement to test each line of criteria one row at a time, quickly testing of thousands of potential parameter sets. Each iteration result is recorded in a table that is queried to retrieve the criteria set that produces the largest number of true positives below a user-specified false discovery rate (FDR). APE then uses these criteria to generate a final results table. The FDR is estimated using the reverse database method on non-redundant datasets (unique peptide identifications), which prevents frequently observed peptides from skewing the error estimate [16].

Figure 1.

Schematic overview of the STEPS methodology. Data can be brought into APE as a flat text file or SQLite database file. Parameter tables are generated within APE using the ranger functionality. The first table defines the parameters to be used and the second table is used to select criteria that correspond to the user-defined FDR, charge state, and NTT. STEPS workflow will output a table of confident peptide identifications, a table of optimal parameters, a crosstab of spectral counting results, and selected QC plots.

In addition to the grid search capability, APE has a number of useful functionalities that make it a powerful data analysis tool, including integrated plotting capabilities. As a result, the STEPS workflow can be configured to generate metric histograms and scatter plots for quality control purposes and to generate optimal filtering parameters (Supplemental Figure 1). Additionally, APE can be configured to produce and export custom tables to facilitate further analysis or for publication. Additional plots and tables can be easily added or subtracted from the workflow. Using SQL Scripting to build workflows provides much faster analyses, elimination of human errors, and gives the ability to work with very large datasets (> 1 million data rows) easily. Moreover, APE maintains a record of all analyses in a time-date stamp fashion that is easily recalled. All analysis steps are visible to the analyst and added comments explain the processing performed for each workflow step (Supplemental Figure 2), which facilitates transparency in the data analysis process. APE effectively creates a detailed lab notebook for every data analysis. The workflow steps are readily modifiable to allow for custom workflows that can be exchanged among researchers. APE also allows for the integration of independent validation metrics, and provides a facile platform for investigating the effects of changes in workflow parameters. Thus, STEPS workflows can be used for “push button” processing, as well as for intensive data processing investigations by expert analysts.

To illustrate its usefulness, we applied the STEPS approach to datasets obtained for a Shewanella oneidensis cell lysate and immunoaffinity-depleted human blood serum [17-18]. Various combinations of filtering parameters were examined to determine discriminative power and to demonstrate the value of exploiting additional information. Spectra for both sample sets had been obtained previously using a custom built four-column high-pressure capillary LC system coupled on-line to an LTQ-Orbitrap mass spectrometer (Thermo Scientific, San Jose CA) via a nanoelectrospray interface manufactured in-house [19]. DeconMSn was utilized to generate .dta files [20], and SEQUEST (Version 27, Revision 12) was used to search all MS/MS data against either an in-house S. oneidensis FASTA protein sequence for MR-1 strain or the human protein database from SwissProt (20,276 entries, released 05/05/2010). FDR was estimated for non-redundant datasets, using a combined forward /reverse database search and the equation (2*Decoy)/(Decoy + Target).

We evaluated the effectiveness of SEQUEST parameters Xcorr, DeltaCn2, and MassDiff, as well as all combinations of these parameters, in terms of numbers of confident peptide identifications. (See Supplemental Table 1 for a summary of ranges and step sizes). Xcorr and DeltaCn2 represent SEQUEST scoring metrics commonly employed for filtering PSMs, and MassDiff is a value that represents the difference between the measured mass and the exact mass of the peptide from the database, which is useful for filtering high mass accuracy data [21]. For this evaluation, we utilized an FDR of 5%. PSMs were partitioned into classes based on their number of tryptic termini (NTT = 0, 1, 2) and charge state (1+, 2+, 3+, 4+, and 5+), which resulted in 15 unique peptide classes (unique combination of charge and NTT) for which an optimal parameter set was generated by STEPS. We also evaluated the effectiveness of incorporating MS-GF [12] values in conjunction with the SEQUEST parameters.

Figure 2 summarizes results obtained for the depleted serum and S. oneidensis datasets with (black bars) and without (grey bars) partitioning, using SEQUEST only parameters (Figure 2A and B) and SEQUEST parameters plus the MS-GF [12] value (Figure 2C and D). Note that for all parameter combinations in Figure 2A and B, partitioning by charge state and NTT provides substantially better differentiation, which agrees with findings in the literature [22]. In some cases without partitioning, single parameter filtering was unable to obtain a 5% FDR. For both datasets, the use of two-metric parameter sets outperformed single-metric filtering, and the utilization of all thee metrics represented the optimum. It is also worthwhile to note that different parameter combinations represent the optimal data model for the different datasets. In Figure 2C and D, incorporation of the MS-GF score provided further increases in the number of peptide identifications at a constant FDR. Interestingly, partitioning by class confers only modest improvements when incorporating the MS-GF score. Additionally, smaller gains are observed when a third parameter is introduced into the model.

Figure 2.

Bar plots depicting the number of confident peptide identifications obtained from A) depleted serum dataset using parameters specified by STEPS optimization, B) S. oneidensis dataset using parameters specified by STEPS optimization, C) serum dataset using STEPS parameters plus MS-GF scoring, and D) S. oneidensis dataset using STEPS parameters plus MS-GF scoring. Results obtained without partitioning peptides into classes are shown in grey and with partitioning are shown in black.

To benchmark performance, we compared results obtained using STEPS with results obtained using PeptideProphet [10], a widely-used PSM filtering algorithm. The comparison was performed using the number of unique peptide identifications at 5% FDR. PeptideProphet probability was used for filtering datasets analyzed with PeptideProphet. Utilization of the STEPS methodology yielded 10.7% increase in true identifications for the S. oneidensis dataset (Figure 3B) and 17.5% increase for the serum dataset (Figure 3A). Significant overlap (Figure 3C and D) was observed between the two filtering approaches. When optimizing parameters, it is always a concern that the parameters will have been over-fit to the specific dataset used to train the parameters. To test for this, the serum dataset (10 LC-MS runs) was randomly divided in 2 parts (5 + 5 LC-MS runs), using one part to train the parameters and the second half as a test set. This analysis was repeated for 4 different partitions of the dataset. On average, only a 2.5% decrease in confident peptide identifications is seen between using the parameters trained on the dataset and applying STEPS parameters to a complimentary dataset, Supplemental Figure 3. This difference is much smaller than the difference between STEPS and PeptideProphet, 17.5% for the serum dataset.

Figure 3.

Comparison of confident peptide identifications obtained using STEPS and PeptideProphet at similar FDR. A) True identifications obtained for A) serum and B) S. oneidensis datasets. Overlap of peptide identifications between STEPS and PeptideProphet for C) depleted serum and D) S. oneidensis datasets.

Inspection of peptides that appeared in STEPS-optimized datasets only (data not shown) revealed peptides that had received low SEQUEST scores, but had high mass measurement accuracy. The majority of these identifications also have confident MS-GF scores, indicating they are legitimate identifications. Tables showing the optimal parameter sets obtained for the depleted serum and cell lysate datasets can be seen in Supplemental Tables 2 and 3, respectively.

To conclude, STEPS methodology represents a conceptually-simple, flexible, and highly discriminative approach to PSM filtering. Implemented using the automated processing engine APE, all data manipulations are visible and modifiable, which allows the analyst to create custom data models. As such, the STEPS methodology can be readily adapted to datasets from any sample type, mass spectrometer, and search engine. By leveraging more of the information available to the analyst, STEPS allows for greater retention of true positive matches from proteomics datasets. This becomes especially significant when filtering PSM's from very complex samples, as in the study of microbial communities[23]. Because of its inherent flexibility and simplicity, STEPS-based methodologies should have widespread applicability in the field.