Skip to main content
PMC home page
American Journal of Human Genetics logoLink to American Journal of Human Genetics
. 2016 Sep 22;99(4):877–885. doi: 10.1016/j.ajhg.2016.08.016

REVEL: An Ensemble Method for Predicting the Pathogenicity of Rare Missense Variants

Nilah M Ioannidis 1,2,34, Joseph H Rothstein 2,3,4,34, Vikas Pejaver 5, Sumit Middha 6, Shannon K McDonnell 7, Saurabh Baheti 7, Anthony Musolf 8, Qing Li 8, Emily Holzinger 8, Danielle Karyadi 9, Lisa A Cannon-Albright 10, Craig C Teerlink 10, Janet L Stanford 11, William B Isaacs 12, Jianfeng Xu 13, Kathleen A Cooney 10,14, Ethan M Lange 15, Johanna Schleutker 16,17, John D Carpten 18, Isaac J Powell 19, Olivier Cussenot 20, Geraldine Cancel-Tassin 20, Graham G Giles 21,22, Robert J MacInnis 21,22, Christiane Maier 23,24, Chih-Lin Hsieh 25, Fredrik Wiklund 26, William J Catalona 27, William D Foulkes 28, Diptasri Mandal 29, Rosalind A Eeles 30, Zsofia Kote-Jarai 30, Carlos D Bustamante 1,31, Daniel J Schaid 7, Trevor Hastie 31,32, Elaine A Ostrander 9, Joan E Bailey-Wilson 8, Predrag Radivojac 5, Stephen N Thibodeau 33, Alice S Whittemore 2,31, Weiva Sieh 2,3,4,∗
PMCID: PMC5065685  PMID: 27666373

Abstract

The vast majority of coding variants are rare, and assessment of the contribution of rare variants to complex traits is hampered by low statistical power and limited functional data. Improved methods for predicting the pathogenicity of rare coding variants are needed to facilitate the discovery of disease variants from exome sequencing studies. We developed REVEL (rare exome variant ensemble learner), an ensemble method for predicting the pathogenicity of missense variants on the basis of individual tools: MutPred, FATHMM, VEST, PolyPhen, SIFT, PROVEAN, MutationAssessor, MutationTaster, LRT, GERP, SiPhy, phyloP, and phastCons. REVEL was trained with recently discovered pathogenic and rare neutral missense variants, excluding those previously used to train its constituent tools. When applied to two independent test sets, REVEL had the best overall performance (p < 10−12) as compared to any individual tool and seven ensemble methods: MetaSVM, MetaLR, KGGSeq, Condel, CADD, DANN, and Eigen. Importantly, REVEL also had the best performance for distinguishing pathogenic from rare neutral variants with allele frequencies <0.5%. The area under the receiver operating characteristic curve (AUC) for REVEL was 0.046–0.182 higher in an independent test set of 935 recent SwissVar disease variants and 123,935 putatively neutral exome sequencing variants and 0.027–0.143 higher in an independent test set of 1,953 pathogenic and 2,406 benign variants recently reported in ClinVar than the AUCs for other ensemble methods. We provide pre-computed REVEL scores for all possible human missense variants to facilitate the identification of pathogenic variants in the sea of rare variants discovered as sequencing studies expand in scale.

Introduction

Interpreting genetic variation from next-generation sequencing (NGS) datasets is essential for the advancement of personalized medicine.1, 2 The vast majority of variants discovered by NGS are rare.3, 4 Recent exome and genome sequencing studies have found that roughly 85% of nonsynonymous variants have alternate allele frequencies (AFs) less than 0.5%, and roughly 100–400 rare nonsynonymous variants are discovered per sequenced individual.3, 4 Rare coding variants play major roles in disease causation and might contribute to the missing heritability from genome-wide association studies.5, 6 However, the majority of nonsynonymous variants discovered by NGS have unknown significance because experimental validation of large numbers of rare variants is infeasible, and association studies require prohibitively large sample sizes to detect rare variants with modest effect sizes with high statistical power. Therefore, computational tools that can accurately predict the pathogenicity of rare variants are needed to help identify those variants that are most likely to cause disease.

Many tools for predicting the pathogenicity of missense variants have been developed based on features such as amino acid or nucleotide conservation and biochemical properties of the amino acid substitutions.7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 However, individual tools often disagree, in part because they utilize different predictive features. Ensemble methods that combine the results of multiple individual predictors can improve performance.19, 20, 21, 22, 23, 24, 25, 26, 27, 28 However, few existing pathogenicity prediction tools have targeted the interpretation of rare variants.24 Current tools are often trained on predominantly common neutral variants and some explicitly impose a minimum AF threshold for defining neutral training variants.15, 21, 25 In contrast, most disease training variants are rare. As a result of this AF imbalance between disease and neutral training variants, tools that rely on AF as a predictive feature might have a lower ability to distinguish disease variants from rare neutral variants than from common ones.24 Biological differences such as higher conservation scores for rare versus common variants might also make rare neutral variants more difficult to distinguish from disease variants.24, 29 Despite the fact that the vast majority of nonsynonymous variants discovered by NGS are rare, the performance of existing prediction tools on rare variants is not well known.30 Thus, there is a growing need for the development and evaluation of tools for predicting the pathogenicity of rare variants.

Here, we present an ensemble method for predicting the pathogenicity of missense variants that outperforms existing approaches overall and when applied to rare variants. The rare exome variant ensemble learner (REVEL) method incorporates recently developed individual prediction tools as features and was trained on recently discovered disease and rare neutral missense variants that did not overlap with the training data for its constituent predictors. We also assembled two large independent test sets of recently discovered pathogenic and benign variants that parallel the likely application of REVEL to newly discovered variants from NGS studies. We benchmark the performance of REVEL and existing ensemble predictors for distinguishing disease mutations from neutral variants across a broad range of AFs. To make our method easily accessible for research and clinical use, we provide pre-computed REVEL scores for all possible human missense variants.31

Material and Methods

Random Forest

We trained a random forest on the set of variants described below by using the R “randomForest” package32 with 1,000 binary classification trees.33, 34 We selected the number of trees to be sufficiently large for the out-of-bag (OOB) error rate to plateau; sensitivity analyses showed that increasing the number of trees to 3,000 did not improve performance on the training dataset. The OOB prediction for a given training variant is the proportion of trees that classified the variant as pathogenic across only those trees in the forest that excluded the variant from their bootstrapped training sample.33 Four features were selected at random as candidates for each split in the random forest trees, which was the default value for 18 features described below. To address the imbalance in the numbers of available disease and neutral training variants, we sampled the same number (n = 6,182) of disease and neutral variants when generating the bootstrapped training set for each tree in the forest. The importance of each predictive feature was measured by the total decrease in the Gini index33 (improvement in node purity) for all splits on that feature, averaged over all trees in the forest.

Training Variants

REVEL was trained with putative rare neutral and disease missense variants. Disease variants were obtained from the Human Gene Mutation Database (HGMD)35 version 2015.2 and were restricted to the set of missense disease mutations (DMs) added to HGMD since August 1, 2012 to minimize overlap with variants previously used to train component features in the REVEL random forest. Missense exome sequencing variants (ESVs) were obtained from the Exome Sequencing Project (ESP)4 European-American and African-American populations, the Atherosclerosis Risk in Communities (ARIC) study36 European-American and African American populations, and the 1000 Genomes Project (KGP)3 European, Yoruban, and Asian populations, as recorded in dbNSFP31 version 2.7. After excluding all disease variants in HGMD and the data sources for test sets 1 and 2 described below, the remaining missense ESVs were considered putatively neutral. For both the disease and neutral training variants, we also excluded all variants that had previously been used to train individual component features in the REVEL random forest; specifically, MutPred,8 PolyPhen-2,10 MutationTaster,11 FATHMM v2.3,14 and VEST 3.0.15 Finally, when a given genetic variant corresponded to multiple amino acid substitutions (AASs) at the protein level, only one AAS was selected at random. After applying all exclusion criteria, a total of 6,182 disease variants and 281,972 putatively neutral ESVs remained. We randomly selected approximately half (n = 140,921) of the putatively neutral ESVs, of which 123,706 rare ESVs (with a maximum alternate AF between 0.1% and 1% across the seven study populations) were used for training, and 17,215 ESVs with an AF >1% were used for initial evaluation of performance across a range of AFs. The remaining half of ESVs were held out for use as independent test variants as described below. Thus, the final training set consisted of 6,182 HGMD disease variants and 123,706 rare neutral ESVs.

Features

REVEL incorporates a total of 18 individual pathogenicity prediction scores from 13 tools as predictive features. MutPred scores were newly computed for this study with the UniProt37 canonical protein sequence when available and the Ensembl38 canonical transcript otherwise. PROVEAN13 scores were obtained from dbNSFP v2.9 (February 3, 2015). 16 additional scores were obtained from dbNSFP v2.7 (September 12, 2014), including eight functional prediction scores (SIFT,7 PolyPhen-2 HVAR and HDIV, LRT,9 MutationTaster, MutationAssessor,12 FATHMM v2.3, and VEST 3.0) and eight conservation scores (GERP++,39 SiPhy,40 three phyloP41 scores for primates, placental mammals, and vertebrates, and three phastCons42 scores for primates, placental mammals, and vertebrates). For PolyPhen-2, FATHMM, and PROVEAN, when multiple protein isoforms were associated with a given variant, we used the average score across all isoforms. Missing features were imputed with the k-nearest neighbors method implemented in the R “impute” package.43 Missing feature values for a given variant were assigned the average value of the non-missing values of that feature from its k = 40 nearest neighboring variants; when more than 50% of features were missing for a given variant, we assigned to each missing feature its overall mean across all variants.

Test Sets

We assembled two independent test sets that did not overlap with either the REVEL training data or the training data for the component features of REVEL. Test set 1 consisted of 935 disease variants added to SwissVar44 (release 2015_10) since August 1, 2012 and approximately half (n = 141,051) of the putatively neutral missense ESVs described above that had not been included in the REVEL training set or initial evaluation. Test set 2 consisted of 1,953 pathogenic or likely pathogenic and 2,406 benign or likely benign variants recently deposited into ClinVar45, 46 by submitters following variant classification guidelines similar to the American College of Medical Genetics and Genomics (ACMG) guidelines.47, 48 Specifically, all single-nucleotide missense variants submitted to ClinVar by GeneDx, Emory Genetics Laboratory, Partners HealthCare Laboratory for Molecular Medicine,49 University of Chicago Genetic Services Laboratory, Ambry Genetics, and Invitae were downloaded on October 13, 2015. We excluded the following from both test sets 1 and 2: all REVEL training variants, all DM variants added to HGMD prior to August 1, 2012, and all variants that had previously been used to train individual component features in REVEL. Finally, to eliminate overlap between the two test sets, we excluded any variants that were present in both SwissVar and ClinVar from test set 1 if benign (n = 9) and from test set 2 if pathogenic (n = 12).

Comparators

We compared the performance of REVEL to seven ensemble prediction tools that were recently developed, widely used, and readily implemented: MetaLR,28 MetaSVM,28 Eigen,50 CADD16 v1.3, DANN,17 Condel,19 and KGGSeq23, 24 v0.8. We ran KGGSeq with the default model selection option that chooses an optimized set of features for each variant.24 We plotted receiver operating characteristic (ROC) curves and compared the area under the ROC curve (AUC) estimates for different tools by using Delong’s test51 implemented in the R “pROC” package.52 We also computed the area under the precision-recall (PR) curve by using the R “ROCR” package.53 For the training variants, REVEL scores were computed by using only the OOB predictions, which have been shown to provide performance estimates that are as accurate as for an independent test set of equal size consisting of variants with similar characteristics.33

Results

Characterization of REVEL Features

The REVEL ensemble score combines pathogenicity predictions from 18 individual scores (features), including 8 conservation scores and 10 functional scores. Figure 1A shows the correlation among individual features. The conservation scores, as well as LRT and Mutation Taster, were almost all highly (Spearman rank correlation coefficient, R > 0.6) to moderately correlated (0.4 < R < 0.6). Five functional scores (MutationAssessor, PROVEAN, VEST, and PolyPhen-2 HDIV and HVAR) were almost all highly correlated. VEST was also highly correlated with several conservation scores, LRT, and MutationTaster. In contrast, FATHMM had low correlation (R < 0.4) with all other scores, and MutPred and SIFT had low to moderate correlation with other scores. The five most important features in the REVEL random forest were FATHMM, VEST, MutationAssessor, MutPred, and PolyPhen-2 HVAR (Figure 1B). The importance measure for an individual feature reflects correlations with other features as well as its intrinsic predictive ability because importance may be shared among correlated features.34

Figure 1.

Figure 1

Open in a new tab

Individual Prediction Tools Included as Features in the REVEL Random Forest

(A) Correlation among the individual features, ordered by hierarchical clustering. The heatmap illustrates the Spearman rank correlation coefficients between features computed for the REVEL training variants.

(B) Relative importance of individual features. Gini importance estimates were normalized to sum to one.

Overall Performance of REVEL as Compared with Other Methods

The REVEL ensemble score discriminated well between HGMD disease mutations and putatively neutral ESVs, and an overall AUC of 0.908 was estimated with OOB predictions for the training set (Figure 2A). The AUC for REVEL was significantly better than any of its constituent features (maximum p < 10−12 for any pairwise comparison), among which VEST (AUC = 0.844) and FATHMM (AUC = 0.824) had the highest AUCs (Table S1). AUCs for the other individual prediction tools ranged from 0.589 to 0.809 and tended to be higher for functional predictors (0.717–0.844) than for conservation scores (0.589–0.791). The AUC for REVEL was also significantly better than the other ensemble methods (maximum p < 10−12 for any pairwise comparison), among which MetaLR (AUC = 0.883) and MetaSVM (AUC = 0.879) had the next highest AUCs (Figure 2A; Table S2).

Figure 2.

Figure 2

Open in a new tab

Performance of Ensemble Methods for Discrimination of Disease Training Variants from Putatively Neutral ESVs

(A) ROC curves for 6,182 HGMD disease mutations and 123,706 rare (AF 0.001–0.01) neutral ESVs used to train REVEL. REVEL scores were computed with only the OOB predictions for its training variants.

(B) AUC for 6,182 HGMD disease mutations and 140,921 neutral ESVs, including REVEL training variants, stratified by neutral variant AF.

Performance for Rare versus Common Neutral Variants

We next compared the performance of REVEL to that of other ensemble methods for discriminating between HGMD disease mutations, which are predominantly rare, and putatively neutral ESVs with AFs ranging from very rare (0.1%–0.3%) to common (>5%). We found that all of the ensemble methods tended to have a worse ability to discriminate disease mutations from rare neutral variants than from common neutral variants (Figure 2B; Table S2). However, compared to other ensemble methods, REVEL had superior discriminatory ability for neutral variants within all AF ranges up to 3%, with the greatest improvements in AUC for rare variants with AF < 0.5% (Figure 2B; Table S2). For neutral variants with AF > 3%, REVEL had the second highest AUC after MetaLR. In addition, the performance of REVEL appeared to be less sensitive to neutral variant AF than other methods. The AUC range for very rare to common variants was narrowest for REVEL (0.897–0.957) and widest for DANN (0.703–0.897), which appeared to be most sensitive to AF (Table S2).

Performance Evaluation in Two Independent Test Sets

In test set 1, consisting of 935 independent disease mutations from SwissVar and 141,051 putatively neutral ESVs, the relative performance of all eight ensemble predictors (Figure 3; Table S3) was similar to that observed in the training set. REVEL had the best performance both overall (p < 10−12) and for neutral variants within all AF ranges up to 5%. For common neutral variants with AF > 5%, REVEL was again surpassed only by MetaLR. The improvement in AUC obtained with REVEL versus with the other ensemble methods was again greatest for rare neutral variants. In test set 2, consisting of 1,953 pathogenic and 2,406 benign variants from ClinVar, we confirmed that REVEL had the best performance among the ensemble methods both overall (p < 10−12) and for neutral variants within all AF ranges up to 3% and that the improvement in AUC was greatest for rare neutral variants (Figure 4; Table S4). All of the ensemble methods had a better overall ability to distinguish benign versus pathogenic variants from ClinVar than putatively neutral ESVs versus disease variants from SwissVar or HGMD, which could be a consequence of the more stringent definition of benign variants from ClinVar. REVEL also had the best overall performance measured by the area under the PR curve (Table S5) across a wide range of proportions of disease variants represented in the training set (4.8%) and test sets 1 (0.7%) and 2 (44.8%).

Figure 3.

Figure 3

Open in a new tab

Performance of Ensemble Methods in an Independent Test Set of SwissVar Disease Mutations and Putatively Neutral ESVs

(A) ROC curves for 935 SwissVar disease mutations and 123,935 rare (AF 0.001–0.01) neutral ESVs that did not overlap with the training set.

(B) AUC for 935 SwissVar disease mutations and 141,051 neutral ESVs, excluding REVEL training variants, stratified by neutral variant AF.

Figure 4.

Figure 4

Open in a new tab

Performance of Ensemble Methods in an Independent Test Set of 1,953 Pathogenic and 2,406 Benign Variants from ClinVar

(A) ROC curves and the AUC for all variants.

(B) AUC for each ensemble method, stratified by neutral variant AF.

Interpretation of REVEL Scores

The REVEL score for an individual variant can range from zero to one, reflecting the proportion of trees in the random forest that classified the variant as pathogenic. REVEL score distributions for the 6,182 HGMD disease and 123,706 putatively neutral ESV training variants, and for all 1,125,160 ESVs reported by ESP, ARIC, and KGP, are shown in Figure 5A. The distributions of REVEL scores were very similar for all reported ESVs and the subset of putatively neutral ESV training variants, with only a small shift toward higher scores for all ESVs. Figure 5B shows the percentiles of the REVEL scores separately for disease and neutral training variants or all ESVs. Figure S1 shows the sensitivity and specificity corresponding to different REVEL score thresholds, above which a variant would be classified as pathogenic. For example, 75.4% of disease mutations but only 10.9% of neutral variants (and 12.4% of all ESVs) have a REVEL score above 0.5, corresponding to a sensitivity of 0.754 and specificity of 0.891. Selecting a more stringent REVEL score threshold of 0.75 would result in higher specificity but lower sensitivity, with 52.1% of disease mutations, 3.3% of neutral variants, and 4.1% of all ESVs being classified as pathogenic.

Figure 5.

Figure 5

Open in a new tab

Interpretation of REVEL Scores

(A) Distribution of REVEL scores for 6,182 disease (red) and 123,706 neutral (blue) training variants and 1,125,160 ESVs (black). REVEL scores were computed with only the OOB predictions for training variants.

(B) Percentiles of the REVEL score distribution for 6,182 disease (red) and 123,706 neutral (blue) training variants and 1,125,160 ESVs (black). REVEL scores were computed with only the OOB predictions for training variants.

Discussion

REVEL is an ensemble method for predicting the pathogenicity of rare missense variants. Rare variants are likely to comprise the vast majority of variants of unknown significance discovered in future sequencing studies. We have shown that REVEL consistently has the best overall performance as compared to existing methods, particularly for distinguishing disease mutations from uncommon neutral missense variants with an AF below 3%. To facilitate use by clinicians and researchers, we have pre-computed REVEL scores for all missense variants in dbNSFP v2.7, a database of all potential nonsynonymous single-nucleotide variants in the human genome. REVEL thus addresses the need for a pathogenicity prediction tool with improved accuracy for interpreting rare genetic variants.

The REVEL method has several strengths. First, REVEL was trained and tested on recently identified disease and neutral variants that could closely resemble novel variants discovered by future NGS studies, which are likely to include variants with lower AFs and more modest effects than previously discovered variants. The REVEL neutral training variants were specifically restricted to AFs between 0.1% and 1% to improve performance when interpreting rare variants. Second, REVEL incorporates a larger number of individual predictors than prior ensemble methods, including both MutPred and VEST, which were among the most important features in the REVEL random forest. MutPred scores, in particular, were not previously widely available and have now been computed for all missense variants in dbNSFP v2.7 as part of this study. Finally, we carefully removed from the training and test sets all variants used to train any of the component predictors in REVEL to reduce overfitting and inflated performance estimates.

A key limitation of this study and others is the reliance on pathogenicity assertions from existing databases, which might be inaccurate and incomplete. Misclassification of training and test variants as disease or neutral would limit both the accuracy of the prediction method and the resulting performance estimates. Nonetheless, we expect that the putative disease variants used to train REVEL are enriched for true disease variants as compared to the putative neutral variants, allowing identification of key predictive features of pathogenic variants. An additional complication is that existing pathogenicity assertions for some variants might have been based in part on predictions from popular tools, such as SIFT and PolyPhen-2, potentially resulting in inflated performance of these predictors and ensemble scores that use them. Finally, the performance of REVEL and other ensemble methods is limited by the accuracy of the component predictors and could benefit from inclusion of additional predictors as they become available in the future.

REVEL had the highest overall performance of any method in independent test sets, although its performance on common variants with AFs > 3%–5% was slightly worse than that of MetaLR or MetaSVM. The strong overall performance of REVEL reflects the fact that the majority of neutral variants in the training and test datasets were rare, as expected for novel variants discovered by NGS. Furthermore, although we carefully removed all variants used to train REVEL and its constituent features from the two test sets, we did not systematically exclude training variants for the comparator ensemble scores; thus, the performance estimates for the comparators could be overly optimistic. MetaLR and MetaSVM had the next highest overall performance. These two ensemble methods included many of the same predictive features as REVEL, except for VEST, MutPred, and PROVEAN, and also included the AF, which could contribute to their greater sensitivity to the neutral variant AF. Condel is a weighted average of FATHMM and MutationAssessor, and its lower performance relative to some ensemble methods could be due to the inclusion of fewer predictive features. Eigen employs an unsupervised approach to separate variants into two classes and also uses fewer predictive features than REVEL, MetaLR, and MetaSVM. CADD and DANN differ from the other ensemble methods in their use of many basic genomic and protein annotations from ENCODE and Ensembl as features in addition to functional predictions from PolyPhen-2 and SIFT. Although CADD and DANN did not perform as well as the other ensemble methods for missense variants, they have important advantages for genome-wide NGS applications because they provide scores for noncoding and regulatory variants that are on the same scale as for coding variants.

The improved performance of REVEL relative to other ensemble methods was greatest for discriminating between disease and rare neutral variants. This result might be partly explained by the fact that REVEL was trained on rare neutral variants with AF < 1% and did not rely on AF as a predictive feature. To our knowledge, one other ensemble predictor, KGGSeq,24 was similarly trained on rare neutral variants. KGGSeq uses many of the same component predictors as REVEL, except for MutPred, and also includes CADD as a predictive feature. However, KGGSeq adaptively selects an optimal subset of features rather than using all features to predict the pathogenicity of each variant, in part to allow exclusion of features with missing data. Possible explanations for the improved performance of REVEL over KGGSeq include use of all features for all variants by first imputing missing scores, importance of MutPred as a predictive feature, and use of a random forest approach rather than logistic regression.

REVEL also outperformed its individual constituent prediction tools, as expected for ensemble methods.19, 20, 21, 22, 23, 24, 25, 26, 27, 28 The top-performing individual tools on our training dataset were VEST,15 FATHMM,14 and MutPred,8 consistent with their high importance in the REVEL random forest. VEST predictions are based on a particularly large set of 86 basic genomic and protein annotations and had the best performance among the individual tools. FATHMM uses a hidden Markov modeling approach to analyze multiple sequence alignments and alignments of conserved protein domain families to compute position-specific amino acid probabilities. The uniqueness of this method might contribute to the low correlation between FATHMM and other prediction tools and high importance in REVEL.28 Finally, the strong performance of MutPred could be because its predictions are based on a particularly detailed model of protein structural and functional properties, including secondary structure, solvent accessibility, functional domains, methylation, phosphorylation, and glycosylation, with quantitative estimates of the probability of losing each property as a result of a particular amino acid change.

In conclusion, REVEL is an ensemble method that outperforms existing tools for distinguishing disease variants from rare neutral variants. REVEL can be used to prioritize the most likely clinically or functionally relevant variants among the sea of rare variants that are increasingly discovered as sequencing studies expand in scale. For example, REVEL scores have been used by the International Consortium of Prostate Cancer Genetics as weights for combining variants discovered by exome sequencing in gene-level case-control studies. Pre-computed REVEL pathogenicity scores for all possible human missense variants, based on GENCODE v9 gene annotations54 for hg19, are available for download (see Web Resources). To aid interpretation, we also provide estimates of REVEL sensitivity and specificity for different score thresholds and the quantiles of the REVEL score in over one million ESVs observed in KGP, ESP, and ARIC. Future studies might explore the application of REVEL to specific genes to evaluate its clinical utility for interpreting variants of unknown significance for a broad spectrum of clinical conditions.

Acknowledgments

This research was funded by NIH grants U01CA089600, R01CA094069, R01LM009722, R01MH105524, K07CA143047, and F32HG008330 and by the Intramural Research Program of the National Human Genome Research Institute, NIH.

Published: September 22, 2016

Footnotes

Supplemental Data include one figure and five tables and can be found with this article online at http://dx.doi.org/10.1016/j.ajhg.2016.08.016.

Web Resources

Supplemental Data

Document S1. Figure S1 and Tables S1–S5
mmc1.pdf (300.8KB, pdf)
Document S2. Article plus Supplemental Data
mmc2.pdf (1.1MB, pdf)

References

  • 1.Peterson T.A., Doughty E., Kann M.G. Towards precision medicine: advances in computational approaches for the analysis of human variants. J. Mol. Biol. 2013;425:4047–4063. doi: 10.1016/j.jmb.2013.08.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Yang Y., Muzny D.M., Reid J.G., Bainbridge M.N., Willis A., Ward P.A., Braxton A., Beuten J., Xia F., Niu Z. Clinical whole-exome sequencing for the diagnosis of mendelian disorders. N. Engl. J. Med. 2013;369:1502–1511. doi: 10.1056/NEJMoa1306555. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Abecasis G.R., Auton A., Brooks L.D., DePristo M.A., Durbin R.M., Handsaker R.E., Kang H.M., Marth G.T., McVean G.A., 1000 Genomes Project Consortium An integrated map of genetic variation from 1,092 human genomes. Nature. 2012;491:56–65. doi: 10.1038/nature11632. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Tennessen J.A., Bigham A.W., O’Connor T.D., Fu W., Kenny E.E., Gravel S., McGee S., Do R., Liu X., Jun G., Broad GO. Seattle GO. NHLBI Exome Sequencing Project Evolution and functional impact of rare coding variation from deep sequencing of human exomes. Science. 2012;337:64–69. doi: 10.1126/science.1219240. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Cirulli E.T., Goldstein D.B. Uncovering the roles of rare variants in common disease through whole-genome sequencing. Nat. Rev. Genet. 2010;11:415–425. doi: 10.1038/nrg2779. [DOI] [PubMed] [Google Scholar]
  • 6.Bamshad M.J., Ng S.B., Bigham A.W., Tabor H.K., Emond M.J., Nickerson D.A., Shendure J. Exome sequencing as a tool for Mendelian disease gene discovery. Nat. Rev. Genet. 2011;12:745–755. doi: 10.1038/nrg3031. [DOI] [PubMed] [Google Scholar]
  • 7.Kumar P., Henikoff S., Ng P.C. Predicting the effects of coding non-synonymous variants on protein function using the SIFT algorithm. Nat. Protoc. 2009;4:1073–1081. doi: 10.1038/nprot.2009.86. [DOI] [PubMed] [Google Scholar]
  • 8.Li B., Krishnan V.G., Mort M.E., Xin F., Kamati K.K., Cooper D.N., Mooney S.D., Radivojac P. Automated inference of molecular mechanisms of disease from amino acid substitutions. Bioinformatics. 2009;25:2744–2750. doi: 10.1093/bioinformatics/btp528. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Chun S., Fay J.C. Identification of deleterious mutations within three human genomes. Genome Res. 2009;19:1553–1561. doi: 10.1101/gr.092619.109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Adzhubei I.A., Schmidt S., Peshkin L., Ramensky V.E., Gerasimova A., Bork P., Kondrashov A.S., Sunyaev S.R. A method and server for predicting damaging missense mutations. Nat. Methods. 2010;7:248–249. doi: 10.1038/nmeth0410-248. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Schwarz J.M., Rödelsperger C., Schuelke M., Seelow D. MutationTaster evaluates disease-causing potential of sequence alterations. Nat. Methods. 2010;7:575–576. doi: 10.1038/nmeth0810-575. [DOI] [PubMed] [Google Scholar]
  • 12.Reva B., Antipin Y., Sander C. Predicting the functional impact of protein mutations: application to cancer genomics. Nucleic Acids Res. 2011;39:e118. doi: 10.1093/nar/gkr407. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Choi Y., Sims G.E., Murphy S., Miller J.R., Chan A.P. Predicting the functional effect of amino acid substitutions and indels. PLoS ONE. 2012;7:e46688. doi: 10.1371/journal.pone.0046688. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Shihab H.A., Gough J., Cooper D.N., Stenson P.D., Barker G.L., Edwards K.J., Day I.N., Gaunt T.R. Predicting the functional, molecular, and phenotypic consequences of amino acid substitutions using hidden Markov models. Hum. Mutat. 2013;34:57–65. doi: 10.1002/humu.22225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Carter H., Douville C., Stenson P.D., Cooper D.N., Karchin R. Identifying Mendelian disease genes with the variant effect scoring tool. BMC Genomics. 2013;14(Suppl 3):S3. doi: 10.1186/1471-2164-14-S3-S3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Kircher M., Witten D.M., Jain P., O’Roak B.J., Cooper G.M., Shendure J. A general framework for estimating the relative pathogenicity of human genetic variants. Nat. Genet. 2014;46:310–315. doi: 10.1038/ng.2892. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Quang D., Chen Y., Xie X. DANN: a deep learning approach for annotating the pathogenicity of genetic variants. Bioinformatics. 2015;31:761–763. doi: 10.1093/bioinformatics/btu703. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Niroula A., Urolagin S., Vihinen M. PON-P2: prediction method for fast and reliable identification of harmful variants. PLoS ONE. 2015;10:e0117380. doi: 10.1371/journal.pone.0117380. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.González-Pérez A., López-Bigas N. Improving the assessment of the outcome of nonsynonymous SNVs with a consensus deleteriousness score, Condel. Am. J. Hum. Genet. 2011;88:440–449. doi: 10.1016/j.ajhg.2011.03.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Crockett D.K., Ridge P.G., Wilson A.R., Lyon E., Williams M.S., Narus S.P., Facelli J.C., Mitchell J.A. Consensus: a framework for evaluation of uncertain gene variants in laboratory test reporting. Genome Med. 2012;4:48. doi: 10.1186/gm347. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Lopes M.C., Joyce C., Ritchie G.R., John S.L., Cunningham F., Asimit J., Zeggini E. A combined functional annotation score for non-synonymous variants. Hum. Hered. 2012;73:47–51. doi: 10.1159/000334984. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Olatubosun A., Väliaho J., Härkönen J., Thusberg J., Vihinen M. PON-P: integrated predictor for pathogenicity of missense variants. Hum. Mutat. 2012;33:1166–1174. doi: 10.1002/humu.22102. [DOI] [PubMed] [Google Scholar]
  • 23.Li M.X., Gui H.S., Kwan J.S., Bao S.Y., Sham P.C. A comprehensive framework for prioritizing variants in exome sequencing studies of Mendelian diseases. Nucleic Acids Res. 2012;40:e53. doi: 10.1093/nar/gkr1257. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Li M.X., Kwan J.S., Bao S.Y., Yang W., Ho S.L., Song Y.Q., Sham P.C. Predicting mendelian disease-causing non-synonymous single nucleotide variants in exome sequencing studies. PLoS Genet. 2013;9:e1003143. doi: 10.1371/journal.pgen.1003143. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Frousios K., Iliopoulos C.S., Schlitt T., Simpson M.A. Predicting the functional consequences of non-synonymous DNA sequence variants--evaluation of bioinformatics tools and development of a consensus strategy. Genomics. 2013;102:223–228. doi: 10.1016/j.ygeno.2013.06.005. [DOI] [PubMed] [Google Scholar]
  • 26.Capriotti E., Altman R.B., Bromberg Y. Collective judgment predicts disease-associated single nucleotide variants. BMC Genomics. 2013;14(Suppl 3):S2. doi: 10.1186/1471-2164-14-S3-S2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Bendl J., Stourac J., Salanda O., Pavelka A., Wieben E.D., Zendulka J., Brezovsky J., Damborsky J. PredictSNP: robust and accurate consensus classifier for prediction of disease-related mutations. PLoS Comput. Biol. 2014;10:e1003440. doi: 10.1371/journal.pcbi.1003440. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Dong C., Wei P., Jian X., Gibbs R., Boerwinkle E., Wang K., Liu X. Comparison and integration of deleteriousness prediction methods for nonsynonymous SNVs in whole exome sequencing studies. Hum. Mol. Genet. 2015;24:2125–2137. doi: 10.1093/hmg/ddu733. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Hodgkinson A., Casals F., Idaghdour Y., Grenier J.C., Hernandez R.D., Awadalla P. Selective constraint, background selection, and mutation accumulation variability within and between human populations. BMC Genomics. 2013;14:495. doi: 10.1186/1471-2164-14-495. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Grimm D.G., Azencott C.A., Aicheler F., Gieraths U., MacArthur D.G., Samocha K.E., Cooper D.N., Stenson P.D., Daly M.J., Smoller J.W. The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Hum. Mutat. 2015;36:513–523. doi: 10.1002/humu.22768. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Liu X., Jian X., Boerwinkle E. dbNSFP v2.0: a database of human non-synonymous SNVs and their functional predictions and annotations. Hum. Mutat. 2013;34:E2393–E2402. doi: 10.1002/humu.22376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Liaw A., Wiener M. Classification and regression by randomForest. R News. 2002;2:18–22. [Google Scholar]
  • 33.Breiman L. Random forests. Mach. Learn. 2001;45:5–32. [Google Scholar]
  • 34.Hastie T., Tibshirani R., Friedman J.H. Springer; 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. [Google Scholar]
  • 35.Stenson P.D., Mort M., Ball E.V., Shaw K., Phillips A., Cooper D.N. The Human Gene Mutation Database: building a comprehensive mutation repository for clinical and molecular genetics, diagnostic testing and personalized genomic medicine. Hum. Genet. 2014;133:1–9. doi: 10.1007/s00439-013-1358-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.The ARIC investigators The Atherosclerosis Risk in Communities (ARIC) Study: design and objectives. The ARIC investigators. Am. J. Epidemiol. 1989;129:687–702. [PubMed] [Google Scholar]
  • 37.Magrane M., UniProt Consortium UniProt Knowledgebase: a hub of integrated protein data. Database (Oxford) 2011;2011:bar009. doi: 10.1093/database/bar009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Flicek P., Amode M.R., Barrell D., Beal K., Billis K., Brent S., Carvalho-Silva D., Clapham P., Coates G., Fitzgerald S. Ensembl 2014. Nucleic Acids Res. 2014;42:D749–D755. doi: 10.1093/nar/gkt1196. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Davydov E.V., Goode D.L., Sirota M., Cooper G.M., Sidow A., Batzoglou S. Identifying a high fraction of the human genome to be under selective constraint using GERP++ PLoS Comput. Biol. 2010;6:e1001025. doi: 10.1371/journal.pcbi.1001025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Garber M., Guttman M., Clamp M., Zody M.C., Friedman N., Xie X. Identifying novel constrained elements by exploiting biased substitution patterns. Bioinformatics. 2009;25:i54–i62. doi: 10.1093/bioinformatics/btp190. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Pollard K.S., Hubisz M.J., Rosenbloom K.R., Siepel A. Detection of nonneutral substitution rates on mammalian phylogenies. Genome Res. 2010;20:110–121. doi: 10.1101/gr.097857.109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Siepel A., Bejerano G., Pedersen J.S., Hinrichs A.S., Hou M., Rosenbloom K., Clawson H., Spieth J., Hillier L.W., Richards S. Evolutionarily conserved elements in vertebrate, insect, worm, and yeast genomes. Genome Res. 2005;15:1034–1050. doi: 10.1101/gr.3715005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Troyanskaya O., Cantor M., Sherlock G., Brown P., Hastie T., Tibshirani R., Botstein D., Altman R.B. Missing value estimation methods for DNA microarrays. Bioinformatics. 2001;17:520–525. doi: 10.1093/bioinformatics/17.6.520. [DOI] [PubMed] [Google Scholar]
  • 44.Mottaz A., David F.P., Veuthey A.L., Yip Y.L. Easy retrieval of single amino-acid polymorphisms and phenotype information using SwissVar. Bioinformatics. 2010;26:851–852. doi: 10.1093/bioinformatics/btq028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Landrum M.J., Lee J.M., Riley G.R., Jang W., Rubinstein W.S., Church D.M., Maglott D.R. ClinVar: public archive of relationships among sequence variation and human phenotype. Nucleic Acids Res. 2014;42:D980–D985. doi: 10.1093/nar/gkt1113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Landrum M.J., Lee J.M., Benson M., Brown G., Chao C., Chitipiralla S., Gu B., Hart J., Hoffman D., Hoover J. ClinVar: public archive of interpretations of clinically relevant variants. Nucleic Acids Res. 2016;44(D1):D862–D868. doi: 10.1093/nar/gkv1222. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Richards C.S., Bale S., Bellissimo D.B., Das S., Grody W.W., Hegde M.R., Lyon E., Ward B.E., Molecular Subcommittee of the ACMG Laboratory Quality Assurance Committee ACMG recommendations for standards for interpretation and reporting of sequence variations: Revisions 2007. Genet. Med. 2008;10:294–300. doi: 10.1097/GIM.0b013e31816b5cae. [DOI] [PubMed] [Google Scholar]
  • 48.Richards S., Aziz N., Bale S., Bick D., Das S., Gastier-Foster J., Grody W.W., Hegde M., Lyon E., Spector E., ACMG Laboratory Quality Assurance Committee Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet. Med. 2015;17:405–424. doi: 10.1038/gim.2015.30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Duzkale H., Shen J., McLaughlin H., Alfares A., Kelly M.A., Pugh T.J., Funke B.H., Rehm H.L., Lebo M.S. A systematic approach to assessing the clinical significance of genetic variants. Clin. Genet. 2013;84:453–463. doi: 10.1111/cge.12257. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50.Ionita-Laza I., McCallum K., Xu B., Buxbaum J.D. A spectral approach integrating functional genomic annotations for coding and noncoding variants. Nat. Genet. 2016;48:214–220. doi: 10.1038/ng.3477. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.DeLong E.R., DeLong D.M., Clarke-Pearson D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–845. [PubMed] [Google Scholar]
  • 52.Robin X., Turck N., Hainard A., Tiberti N., Lisacek F., Sanchez J.C., Müller M. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. 2011;12:77. doi: 10.1186/1471-2105-12-77. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Sing T., Sander O., Beerenwinkel N., Lengauer T. ROCR: visualizing classifier performance in R. Bioinformatics. 2005;21:3940–3941. doi: 10.1093/bioinformatics/bti623. [DOI] [PubMed] [Google Scholar]
  • 54.Harrow J., Frankish A., Gonzalez J.M., Tapanari E., Diekhans M., Kokocinski F., Aken B.L., Barrell D., Zadissa A., Searle S. GENCODE: the reference human genome annotation for The ENCODE Project. Genome Res. 2012;22:1760–1774. doi: 10.1101/gr.135350.111. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Document S1. Figure S1 and Tables S1–S5
mmc1.pdf (300.8KB, pdf)
Document S2. Article plus Supplemental Data
mmc2.pdf (1.1MB, pdf)

Articles from American Journal of Human Genetics are provided here courtesy of American Society of Human Genetics

RESOURCES