A multi-dimensional integrative scoring framework for predicting functional variants in the human genome

Summary

Attempts to identify and prioritize functional DNA elements in coding and non-coding regions, particularly through use of in silico functional annotation data, continue to increase in popularity. However, specific functional roles can vary widely from one variant to another, making it challenging to summarize different aspects of variant function with a one-dimensional rating. Here we propose multi-dimensional annotation-class integrative estimation (MACIE), an unsupervised multivariate mixed-model framework capable of integrating annotations of diverse origin to assess multi-dimensional functional roles for both coding and non-coding variants. Unlike existing one-dimensional scoring methods, MACIE views variant functionality as a composite attribute encompassing multiple characteristics and estimates the joint posterior functional probabilities of each genomic position. This estimate offers more comprehensive and interpretable information in the presence of multiple aspects of functionality. Applied to a variety of independent coding and non-coding datasets, MACIE demonstrates powerful and robust performance in discriminating between functional and non-functional variants. We also show an application of MACIE to fine-mapping and heritability enrichment analysis by using the lipids GWAS summary statistics data from the European Network for Genetic and Genomic Epidemiology Consortium.

Introduction

Ever since the completion of the human genome sequence, substantial effort has been invested into identifying and annotating functional DNA elements. For any given genetic variant, a diverse set of functional annotations is now available. For example, the computational tool PolyPhen¹ predicts damaging effects of missense mutations. PhastCons,² PhyloP,³ and GERP++⁴ leverage comparative sequence information to identify regions that show evolutionary conservation. The Encyclopedia of DNA Elements (ENCODE) has extensively mapped regions of transcription-factor binding, chromatin structure, and histone modification and has effectively assigned biochemical functions for ∼80% of the genome.⁵ Other initiatives such as the Roadmap Epigenomics project⁶ and FANTOM5 project^7,8 also provide evidence for potential regulatory variants in the human genome.

Although functional annotations vary considerably with respect to the specific elements they evaluate and the extent of the human genome they annotate, it is well understood that they provide complementary lines of evidence.⁹ Therefore, if researchers are to obtain a comprehensive understanding of the biological relevance of genomic segments, all of the information provided by different annotations should be jointly synthesized. However, it remains challenging to summarize these diverse functional annotations in an insightful and interpretable manner.

Current algorithmic scoring frameworks utilize a variety of statistical and machine-learning methods to aggregate information from large, diverse sets of individual annotations into single measures of functional importance. Supervised tools such as CADD,¹⁰ DANN,¹¹ GWAVA,¹² FATHMM-MKL,¹³ and FATHMM-XF¹⁴ build machine-learning classifiers on training sets with pre-labeled functional statuses, e.g., fine-mapped pathogenic or disease-associated variants labeled against benign or neutral variants. Such supervised approaches rely strongly on the quality of labels in the training set. Therefore, they might demonstrate suboptimal performance when inaccurate or biased labels are used. Unsupervised methods such as EIGEN,¹⁵ GenoCanyon,¹⁶ PINES,¹⁷ and FUN-LDA¹⁸ do not rely on any labeled training data. They possess advantages in studying non-coding regions, where our current lack of knowledge often precludes gold-standard training data labels. A third group of methods including fitCons¹⁹ and LINSIGHT²⁰ use evolution-based approaches that characterize the potential effect of natural selection at each genomic location by using polymorphism and divergence data. Recent reviews provide a more detailed discussion of available functional annotation tools.21, 22, 23

Although existing methods attempt to integrate functional annotations through various approaches, to the best of our knowledge, these methods all summarize the annotation information with a single rating. In doing so, they implicitly assume that variant function can be described along a single axis, whereby variants are more functional at one end of the axis and less functional at the other end. This assumption might be reasonable if interest lies in predicting a specific aspect of variant function (e.g., regulatory behavior) and if all annotations used as input are intended to predict that same aspect. However, if multiple aspects of variant function are simultaneously of interest, then it is unclear how to interpret the one-dimensional consolidation of annotations measuring different aspects of function, especially when these annotations appear to provide orthogonal information, e.g., weak correlation between evolutionary conservation scores and regulatory scores (Figure 1). Therefore, it is of interest to construct multi-dimensional integrative scores capable of capturing multiple facets of variant function simultaneously.

Figure 1.

Heatmap demonstrating the correlation between individual and integrative functional scores for ClinVar pathogenic and benign non-coding variants

In this paper, we propose multi-dimensional annotation-class integrative estimation (MACIE), an unsupervised multivariate mixed-model framework capable of synthesizing multiple categories of annotations and producing interpretable multi-dimensional integrative scores (Figure S1). Instead of a single rating, MACIE explicitly defines variant function as a vector of binary outcomes, each of which captures functionality corresponding to a specific class of annotations. Correlations within and between the different classes of annotations are explicitly modeled, another advancement over existing methods. Using the expectation-maximization algorithm, MACIE calculates the joint posterior probabilities that a genomic position is functional (material and methods).

Because of its multivariate formulation, MACIE is able to provide detailed and nuanced assessments of variant functionality. Output from MACIE is highly interpretable as a result of the specificity allowed by multiple functional classes. Additionally, the MACIE framework allows for considerable versatility to incorporate data in a manner that is most biologically relevant to the scientific question of interest. We apply MACIE to multiple independent coding and non-coding testing sets and show that, compared to existing integrative scores, MACIE scores consistently provide robust and best or near-best performance in discriminating between functional and non-functional variants.

Material and methods

The MACIE generalized linear mixed model (GLMM)

Suppose there are $N$ genetic variants in total and we are interested in $M$ latent annotation classes, each containing $L_j$ annotation scores. For example, the first class might consist of $L_1=4$ protein functional scores, and the second class might consist of $L_2=8$ evolutionary conservation scores. For genetic variant $i$ and annotation class $j$, we denote the set of $L_j$ annotations as $y_{ij}={\left(y_{ij1},\dots ,y_{ij{L}_j}\right)}^T$, such that each variant is described by $L=\sum _{j=1}^M{L}_j$ annotations in total. We are interested in estimating for each variant $i$ the vector of binary functional statuses ${\mathit{c}}_i=\left(c_{i1},\dots c_{iM}\right)$, where $c_{ij}$ is the unobserved latent functional status for class $j$. Continuing our example, $c_{i1}$ would denote membership in the evolutionarily conserved function class, and $c_{i2}$ would denote membership in the regulatory function class. Conditional on $c_{ij}$ and a random effect variable $b_{ijk}$, we assume that the elements of ${\mathit{y}}_{ij}$ are independent observations, each following a GLMM. That is, for $j=1,\dots ,M$ and $k=1,\dots ,L_j$,

$$g_{jk}\left(E\left(y_{ijk}|c_{ij},b_{ijk}\right)\right)={\beta }_{0jk}+{\beta }_{1jk}{c}_{ij}+b_{ijk},$$

where additional correlations between elements of ${\mathit{y}}_{ij}$ are allowed if we assume that

$${\mathit{b}}_{ij}=\left(\begin{array}{l}{b}_{ij1}\\ ⋮\\ b_{ij{L}_j}\end{array}\right)\stackrel{iid}{˜}MVN\left(\mathbf{0},{\mathbf{\text{Σ}}}_j\left(\mathit{\theta }\right)\right).$$

The MACIE score for a given genetic variant $i$ is defined by $p\left({\mathit{c}}_i|{\mathit{y}}_i\right)$, that is, the posterior probability vector of the unobserved class label ${\mathit{c}}_i$, conditional on the observed annotations ${\mathit{y}}_i$. Because of thconditional independence of ${\mathit{y}}_i$ given ${\mathit{c}}_i$ and ${\mathit{b}}_i$ (the collections of ${\mathit{y}}_{ij}$, $c_{ij}$, and ${\mathit{b}}_{ij}$, respectively, for $j=1,\dots ,M$), an expectation-maximization (EM) algorithm provides a natural approach.²⁴

The expectation-maximization algorithm

The complete-data log-likelihood of $\left(\mathit{y},\mathit{c},\mathit{b}\right)={\left\{{\mathit{y}}_i,{\mathit{c}}_i,{\mathit{b}}_i\right\}}_{i=1}^N$ is given by

$$\log f\left(\mathit{y},\mathit{c},\mathit{b}\right)=\sum _{i=1}^N\left(\sum _{j=1}^M\sum _{k=1}^{L_j}\log f_{jk}\left(y_{ijk}|c_{ij},b_{ijk};{\mathit{\beta }}_{jk},{\phi }_{jk}\right)+\sum _{j=1}^M\log f\left({\mathit{b}}_{ij};\mathit{\theta }\right)+\log p\left({\mathit{c}}_i;\mathit{\gamma }\right)\right)$$

where $\mathit{\beta },\mathit{\phi },\mathit{\theta },\text{and}\mathit{\gamma }$ are (unknown) model parameters. Because both $\mathit{c}$ and $\mathit{b}$ are unobserved, we proceed with the following EM algorithm:

• 1.Initiate reasonable parameter values. Denote the parameter estimates at iteration r by $\left({\hat{\mathit{\beta }}}_{jk}^{\left(r\right)},{\hat{\phi }}_{jk}^{\left(r\right)},{\hat{\mathit{\theta }}}^{\left(r\right)},{\hat{\mathit{\gamma }}}^{\left(r\right)}\right)$.
• 2.(E-step a) Compute the posterior distribution ${\hat{f}}^{\left(r\right)}\left({\mathit{c}}_i,{\mathit{b}}_i|{\mathit{y}}_i\right)={\hat{f}}^{\left(r\right)}\left({\mathit{b}}_i|{\mathit{y}}_i,{\mathit{c}}_i\right){\hat{p}}^{\left(r\right)}\left({\mathit{c}}_i|{\mathit{y}}_i\right)$ via Bayes’ theorem:

$$f\left({\mathit{b}}_i|{\mathit{y}}_i,{\mathit{c}}_i\right)=\prod _{j=1}^M\frac{f\left({\mathit{y}}_{ij}|c_{ij},{\mathit{b}}_{ij}\right)f\left({\mathit{b}}_{ij}\right)}{\int f\left({\mathit{y}}_{ij}|c_{ij},{\mathit{b}}_{ij}\right)f\left({\mathit{b}}_{ij}\right)\text{d}{\mathit{b}}_{ij}}$$

$$p\left({\mathit{c}}_i|{\mathit{y}}_i\right)=\frac{p\left({\mathit{c}}_i,{\mathit{y}}_i\right)}{p\left({\mathit{y}}_i\right)}=\frac{{\prod }_{j=1}^M\left[\int f\left({\mathit{y}}_{ij}|c_{ij},{\mathit{b}}_{ij}\right)f\left({\mathit{b}}_{ij}\right)\text{d}{\mathit{b}}_{ij}\right]\cdot p\left({\mathit{c}}_i\right)}{{\sum }_{\mathit{c}\in {\left\{\mathrm{0,1}\right\}}^M}{\prod }_{j=1}^M\left[\int f\left({\mathit{y}}_{ij}|{\mathit{c}}_{ij},{\mathit{b}}_{ij}\right)f\left({\mathit{b}}_{ij}\right)\text{d}{\mathit{b}}_{ij}\right]\cdot p\left(\mathit{c}\right)}$$

• 3.(E-step b) Compute expected score functions with respect to the posterior distribution of $f\left({\mathit{c}}_i,{\mathit{b}}_i|{\mathit{y}}_i\right)$, i.e., $E_{\mathit{c},\mathit{b}}S\left({\mathit{\beta }}_{jk}\right),E_{\mathit{c},\mathit{b}}S\left({\phi }_{jk}\right),E_{\mathit{c},\mathit{b}}S\left(\mathit{\theta }\right),E_{\mathit{c},\mathit{b}}S\left(\mathit{\gamma }\right)$, where $S\left(\left\{{\mathit{\beta }}_{jk},{\phi }_{jk},\mathit{\theta },\mathit{\gamma }\right\}\right)=\partial \log f\left(\mathit{y},\mathit{c},\mathit{b}\right)/\partial \left\{{\mathit{\beta }}_{jk},{\phi }_{jk},\mathit{\theta },\mathit{\gamma }\right\}$ are the complete data score functions of ${\mathit{\beta }}_{jk},{\phi }_{jk},\mathit{\theta },\mathit{\gamma }$, respectively.
• 4.(M-step) Update $\left({\hat{\mathit{\beta }}}_{jk}^{\left(r+1\right)},{\hat{\phi }}_{jk}^{\left(r+1\right)},{\stackrel{\mathbf{ˆ}}{\mathbf{\theta }}}^{\left(r+1\right)},{\stackrel{\mathbf{ˆ}}{\mathbf{\gamma }}}^{\left(r+1\right)}\right)$ by solving the expected score equations from step 3.
• 5.Iterate between steps 2–4 until convergence of all parameters.

To select initial parameter values, for each functional class $j$, we first perform a k-means clustering algorithm with $k=2$ to classify the functional status $\left(c_{ij}\right)$ of each variant $i$. We then fit a linear or logistic regression model for each individual annotation score used in this class and obtained the fitted parameter estimates as initial parameter values. We found that our results are robust to choices of initial values. The EM algorithm proceeds until the relative changes in all estimated parameters are sufficiently small (<10⁻⁴) with a maximum of 200 iterations. The final converged estimate $\hat{p}\left({\mathit{c}}_i|{\mathit{y}}_i\right)$ (output of step 2) corresponds to the MACIE score for genetic variant $i$. Given the fitted model parameters and the full set of annotation scores for a new genetic variant $i^{\text{′}}$, the MACIE score of variant $i^{\text{′}}$ is defined as the (predicted) posterior probability vector $\hat{p}\left({\mathit{c}}_{i^{\text{′}}}=\mathit{z}|{\mathit{y}}_{i^{\text{′}}}\right),\mathit{z}\in {\left\{\mathrm{0,1}\right\}}^M$. It can be calculated with one additional iteration of the EM algorithm. Full details of the EM-algorithm derivation are provided in the supplemental material and methods.

Construction of MACIE training sets

We used the proposed framework to fit the MACIE GLMM models and compute MACIE scores for (1) non-synonymous coding and (2) non-coding and synonymous coding variants separately because the two types of variants are expected to have highly different functional profiles.¹⁵ All non-synonymous coding annotations and non-coding and synonymous coding evolutionary conservation annotations were downloaded from EIGEN. The non-coding and synonymous coding epigenetic annotations were downloaded from CADD database¹⁰ v1.3.

Non-synonymous coding variants

For the non-synonymous coding training set, we randomly extracted 10% of the variants with a match in the dbNSFP database.²⁵ This database excludes synonymous variants that fall in coding regions but do not alter protein function. Only one unique variant per position was selected, and variants residing on sex chromosomes X and Y were removed so that potential sources of bias or lack of annotations were mitigated. The final set included approximately 2.2 million variants. For each variant in the training set, 4 protein substitution damage scores (SIFT;²⁶ PolyPhenDiv and PolyPhenVar;¹ and Mutation Assessor²⁷) and eight evolutionary conservation scores (GERP_NR and GERP_RS;⁴ PhyloP primate (PhyloPri), placental mammal (PhyloPla), and vertebrate (PhyloVer);³ and PhastCons primate (PhastPri), placental mammal (PhastPla), and vertebrate (PhastVer)²) were extracted from the EIGEN database.¹⁵ Thus, we defined the two-class MACIE model $\left(M=2\right)$ for non-synonymous coding variants to assess damaging protein-coding function and evolutionarily conserved function. Full information on the MACIE model for non-synonymous coding variants and the corresponding MACIE GLMM regression parameter estimates from the training set are presented in the supplemental material and methods and Tables S1 and S2. We used this model to compute the MACIE scores for all non-synonymous variants in the human genome.

Non-coding and synonymous coding variants

For the non-coding and synonymous coding training set, we extracted a random sample comprising 10% of the 1000 Genomes Project variants that were located within 500 bp upstream of a gene start site and did not possess a match in dbNSFP. We again removed duplicated variants with multiple alternative alleles and variants on sex chromosomes X and Y to mitigate potential bias. The final training set included 36,431 variants. For each variant in the training set, the same eight evolutionary conservation scores used for coding variants were extracted from the EIGEN database.¹⁵ A total of 28 transformed epigenetic scores were additionally extracted from the CADD database¹⁰ v.1.3; these included including three histone marks and 12 open chromatin marks from the ENCODE Project;⁵ three transcription factor binding-site scores; GC content, CpG content; five chromatin-state probabilities derived from the 15-state ChromHMM model;²⁸ a background selection score;²⁹ and two physical-distance metrics.¹⁰ Thus, we defined the two-class MACIE model $\left(M=2\right)$ for non-coding and synonymous coding variants to assess evolutionarily conserved function and epigenetic regulatory function. Full information on the MACIE model for non-coding and synonymous coding variants and the corresponding MACIE GLMM regression parameter estimates from the training set are presented in the supplemental material and methods and Tables S1, S3, and S4. We used this model to compute the MACIE scores for all non-coding and synonymous coding variants in the human genome.

Results

Benchmarking the performance of MACIE with that of other integrative scoring methods

We compared the predictive performance of MACIE against existing variant classifiers, including CADD,¹⁰ FATHMM-XF,¹⁴ EIGEN,¹⁵ fitCons,¹⁹ LINSIGHT,²⁰ and DANN¹¹ over a range of realistic variant-assessment scenarios. Specifically, we assessed the ability of each score to identify clinically significant variants from ClinVar;^30,31 loss-of-function variants in the BRCA1 (MIM: 113705) gene uncovered through saturation genome editing (SGE);³² promoters and enhancers from the FANTOM5 project defined by cap analysis of gene expression (CAGE);^7,8 and experimentally verified functional variants from massive parallel reporter assays (MPRAs).^33,34 Some alternative scoring methods were excluded because of difficulties related to providing a proper comparison of results. For example, LINSIGHT is designed to predict the deleteriousness of non-coding variants, so we did not include it in the comparison for non-synonymous coding variants.

Distribution of posterior probabilities for non-coding and synonymous coding variants in the training set

In Table S5 we provide the posterior probabilities of each functional class averaged across all the non-coding and synonymous coding variants in the training set. The predicted MACIE score for a given variant can be interpreted as the posterior probability that the variant belongs to (0,0), neither conserved nor regulatory classes; (1,0), the conserved but not the regulatory class; (0,1), the regulatory but not the conserved class; and (1,1), both the conserved and the regulatory classes. The four MACIE scores necessarily sum up to 1. A chi-square test comparing observed and expected percentages under independence of evolutionary conservation and regulatory classes gives a significant p value of less than $2.2\times {10}^{-16}$, suggesting that the two classes are correlated. Because the observed percentage of functional variants that belong to (1,1) is statistically significantly greater than the expected percentage under independence (3.18% > 1.92%), we find strong evidence of enrichment of regulatory activity in conserved regions.

ClinVar pathogenic and benign variants

We first validated our method on a testing set consisting of all variants recorded in the ClinVar database.^30,31 We extracted variant effect predictor (VEP) information from GENCODE³⁵ and used it to separate non-synonymous coding variants from non-coding and synonymous coding variants in ClinVar. We then applied the two MACIE models described above to the respective partitions. We combined the ClinVar categories “pathogenic” and “likely pathogenic” into a single pathogenic class and treated these variants as the putatively functional class. Similarly, we combined the ClinVar categories “benign” and “likely benign” into a single benign class and treated these variants as the putatively non-functional class. The remaining variants were categorized as having uncertain significance.

We first tested MACIE’s ability to distinguish pathogenic variants $\left(n=\mathrm{33,714}\right)$ from their benign counterparts $\left(n=\mathrm{14,410}\right)$ among ClinVar non-synonymous variants through two approaches. First, we calculated two marginal MACIE scores: (1) MACIE-damaging protein function score (denoted by MACIE-protein) as the sum of the posterior probabilities of “damaging protein functional/not conserved” and “damaging protein functional/conserved”; and (2) MACIE-conserved score as the sum of the posterior probabilities of “damaging protein functional/conserved” and “not damaging protein functional/conserved.” We also considered the posterior probability of either “damaging protein functional/conserved” (denoted by MACIE-anyclass) by summing the posterior probabilities corresponding to at least one functional class (Table S6). This example illustrates the versatility of MACIE’s posterior probability outputs, which can be summed to form new probability measures with various informative interpretations depending on the specific needs of each analysis.

Figure 2 provides the receiver operating characteristic (ROC) curves and area under the curves (AUC) for the three MACIE approaches and seven one-dimensional scores for ClinVar non-synonymous variants. Of the methods considered, MACIE-damaging protein function score delivered the highest discrimination power (AUC = 0.93), followed by CADD (AUC = 0.91), EIGEN (AUC = 0.90), and MACIE-anyclass (AUC = 0.89). These four methods substantially outperformed the supervised DANN (AUC = 0.78), the supervised FATHMM-XF (AUC = 0.74), and the evolution-based fitCons (AUC = 0.54). We observed similar results when we distinguished between pathogenic missense (as opposed to all non-synonymous) variants $\left(n=\mathrm{21,409}\right)$ and their benign counterparts $\left(n=\mathrm{14,035}\right)$ in ClinVar (Figure S2).

Figure 2.

ROC curves comparing the performances of MACIE and other functional scores in discriminating between ClinVar pathogenic and benign non-synonymous coding variants

Next, we identified 40,109 non-coding variants, including 6,551 pathogenic variants and 33,558 benign variants, from the ClinVar database in total. For these non-coding variants, we chose to calculate a marginal MACIE-conserved score because ClinVar pathogenic non-coding variant labels track closely with evolutionary conservation scores (Figure 1). ROC curves and AUCs for discriminating between the pathogenic and benign variants are provided in Figure S3. The MACIE-conserved score showed comparable performance (AUC = 0.95) to the FATHMM-XF score, which showed the highest discrimination power (AUC = 0.97). The outperformance of FATHMM-XF in this specific example should be expected because FATHMM-XF is a supervised machine-learning method trained on labels that bear many similarities to the labels defined in ClinVar, whereas MACIE is an unsupervised method. The MACIE-conserved score substantially outperformed the unsupervised method EIGEN (AUC = 0.84) and the evolution-based method fitCons (AUC = 0.55).

Loss-of-function non-synonymous coding variants in BRCA1

We evaluated MACIE’s performance in predicting the deleteriousness of non-synonymous coding variants located within 13 exons that encode functionally critical domains of BRCA1. We fit based a two-component Gaussian mixture model on the saturation genome editing function scores to classify all BRCA1 variants as loss-of-function (LOF), intermediate (INT), or functional (FUNC) in a decreasing order of severity.³² Thus, FUNC corresponds to benign variants in this experiment. We selected reported LOF non-synonymous coding variants $\left(n=674\right)$ as the putative functional set and designated FUNC non-synonymous coding variants $\left(n=\mathrm{1,443}\right)$ as the putative non-functional set. Among all the methods compared (Figure 3), the MACIE-damaging protein function score showed the highest predictive power (AUC = 0.91), followed by EIGEN (AUC = 0.88) and MACIE-anyclass (AUC = 0.88). The top three scores were much more powerful than CADD (AUC = 0.78), FATHMM-XF (AUC = 0.69), DANN (AUC = 0.60), and fitCons (AUC = 0.42). We observed similar results when distinguishing between BRCA1 LOF non-synonymous coding variants $\left(n=674\right)$ and ClinVar benign non-synonymous coding variants $\left(n=\mathrm{14,410}\right)$ (Figure S4).

Figure 3.

Performance comparing MACIE and other functional scores on BRCA1 non-synonymous coding variants

ROC curves comparing the performances of MACIE and other functional scores in discriminating between loss-of-function (LOF) and functional (FUNC) non-synonymous coding variants within 13 exons that encode functionally critical domains of BRCA1 on the basis of saturation genome editing (SGE) data. Here the LOF class is our putative functional class, and the FUNC class is our putative non-functional class.

FANTOM5 CAGE-defined promoters and enhancers among 1000 genomes non-coding variants

We tested the ability of MACIE to identify promoter regions defined by the FANTOM5 project’s cap analysis of gene expression.^7,8 A total of 110,895 out of approximately 80 million non-coding variants from the 1000 Genomes Project phase 3 data³⁶ were mapped to such regions and therefore labeled as CAGE promoters. For each identified CAGE promoter variant, we used the 1000 Genomes Project database to randomly select a matched control variant (non-promoter) that possessed the same minor-allele frequency (MAF) and same minimum distance to any gene transcription start site that was located at least 500 kilobase (kb) away from the promoter variant, yielding a total number of 97,298 variants in the control set (it was not possible to find a matched control for each CAGE variant). Similar to the previous analysis, we calculated a marginal MACIE-regulatory score by summing the two probabilities corresponding to the regulatory class (denoted by MACIE-regulatory). ROC curves and AUCs for discriminating between CAGE promoters and non-promoters are provided in Figure 4A. MACIE-regulatory and MACIE-anyclass scores showed the highest discrimination power (AUC = 0.75), followed by EIGEN with AUC = 0.74. FATHMM-XF (AUC = 0.54) and fitCons (AUC = 0.56) scores performed poorly because these one-dimensional scores are unable to capture epigenetic functionality. We also performed a similar analysis by contrasting CAGE-identified enhancers $\left(n=\mathrm{520,987}\right)$ versus non-enhancers $\left(n=\mathrm{448,253}\right)$ by using non-coding variants from the 1000 Genomes Project. The results were similar, and MACIE-regulatory score displayed the highest predictive power and significantly outperformed all other existing methods (Figure 4B).

Figure 4.

Performance comparing MACIE and other functional scores on predicting CAGE promoters and CAGE enhhancers from 1000 Genomes Project phase 3 data

ROC curves comparing the performances of MACIE and other functional scores in discriminating between (A) CAGE-identified promoters and non-promoters and (B) CAGE-identified enhancers and non-enhancers among non-coding variants from 1000 Genomes Project phase 3 data. For CAGE Enhancer predictions, LINSIGHT was excluded because it uses the FANTOM5 enhancer label as one of the genomic features in building the LINSIGHT score.

MPRA-validated variants and dsQTLs in lymphoblastoid cell lines

We used test sets from the massively parallel reporter assay to examine the performance of MACIE for predicting cell type- and tissue-specific regulatory variants. The MPRA dataset included validated regulatory variants in lymphoblastoid cell lines (LCLs).³³ We paired each positive variant $\left(n=693\right)$ with four control variants from MPRA when neither allele showed significant differential expression at a Bonferroni-corrected p value threshold of 0.1 $\left(n=\mathrm{2,772}\right)$.³⁷ Figure 5A shows that the MACIE-regulatory score produced the highest discrimination power (AUC = 0.68); the second-best performing method was LINSIGHT (AUC = 0.64).

Figure 5.

Performance comparing MACIE and other functional scores on predicting regulatory variants in LCLs against control variants

ROC curves comparing the performances of MACIE and other functional scores for the prediction of (A) validated regulatory variants in lymphoblastoid cell lines (LCLs) from massively parallel reporter assays (MPRAs) and (B) dsQTLs identified via DNase I sequencing data in LCLs against control variants.

Finally, we evaluated the performance of our proposed method on a collection of dsQTLs that were identified via DNase I sequencing data from human lymphoblastoid cell lines.³⁸ Variants possessing association p values less than $1\times {10}^{-5}$ and residing within 100 bp of their corresponding DNase I-hypersensitive sites were chosen as the putatively functional set $\left(n=560\right)$.³⁹ The control set of variants was randomly selected from a larger set of common variants (MAF > 5%) falling in the top 5% of DNase I sensitivity sites used for identifying dsQTLs in the original study $\left(n=\mathrm{2,240}\right)$. We observed that the MACIE-regulatory score exhibited a larger AUC (AUC = 0.76) than all other methods (Figure 5B). The MACIE-anyclass score also delivered robust performance on MPRA-validated and dsQTL datasets.

In summary, MACIE consistently ranked as one of the most powerful, robust and interpretable functional annotation intergrative score methods across a variety of settings and scientific questions. Given that all the scoring methods in the benchmarking comparisons used a similar set of annotations, the better performance of MACIE across a variety of testing sets shows that whereas one-dimensional scores have gaps in functional profile coverage, a multi-dimensional scoring method offers robust and interpretable predictive performance. The ability of MACIE to interrogate variant functionality from multiple perspectives, at a level that is highly competitive with or better than state-of-the-art methods, is unmatched by existing integrative functional scoring methods.

MACIE prioritizes functional variants by using lipid GWAS data

To illustrate the utility of MACIE scores in identifying plausible functional causal variants in genetic association studies, we applied MACIE to the publicly available lipid GWAS data from the European Network for Genetic and Genomic Epidemiology (ENGAGE) Consortium.⁴⁰ This dataset consists of lipid GWAS summary statistics for 9.6 million single-nucleotide variants (SNVs) across 62,166 samples (Table S7). We focused on genome-wide significant $\left(p<5\times {10}^{-8}\right)$ SNVs associated with low-density lipoprotein cholesterol (LDL-C), high-density lipoprotein cholesterol (HDL-C), triglycerides (TG), and total cholesterol (TC). In total, for non-synonymous coding SNVs, we found 8, 9, 5, and 10 SNVs that were predicted to belong to the protein-damaging class with probability greater than 0.9 for LDL-C, HDL-C, TG, and TC, respectively. For non-coding and synonymous coding SNVs, 643, 377, 322, and 850 SNVs were predicted to belong to the regulatory class with probability greater than 0.9, and 9, 8, 10, and 12 SNVs were predicted to belong to both the evolutionarily conserved and the regulatory class with probability greater than 0.9 for LDL-C, HDL-C, TG, and TC, respectively. Among all SNVs, 50, 64, 39, and 61 SNVs were predicted to belong to the evolutionarily conserved class with probability greater than 0.9 for LDL-C, HDL-C, TG, and TC, respectively (Tables S8–S15). Compared to the total number of marginally significant SNVs for each trait (Table S7), the MACIE scores reduce the number of SNVs prioritized for follow-up by an order of magnitude, saving much cost and effort in effectively pinpointing SNVs with relevant biological function.

For example, for LDL-C, rs7412 (chr19: 45412079 C/T; $p<1\times {10}^{-316}$) was the most significant SNV in APOE (MIM: 107741). Because both MACIE-protein and MACIE-conserved scores provided a prediction greater than 0.95, we predicted this known common missense SNV to be functional. These predictions highlight the multiple functional roles of this SNV. It is also worth noting that the second-most-significant SNV rs1065853 (chr19: 45413233 G/T; $p<1\times {10}^{-316}$) is in extremely high linkage disequilibrium (LD) with the leading SNV rs7412 $\left({\text{r}}^2>0.8\right)$(Figure 6). MACIE scores indicate that rs1065853 (upstream variant of APOC1 [MIM: 107710]) might possess a regulatory role; its MACIE-regulatory score is greater than 0.99, possibly suggesting that both the missense and regulatory variants can be putatively causal in affecting LDL-C concentrations. Similar results were observed for TC (Figure S5). For HDL-C, although the single most significant SNV was rs17231506 (chr16: 56994528 C/T; $p=6.88\times {10}^{-316}$), the MACIE prediction was less than 0.01 for both classes. By scanning across the CETP (MIM: 118470) locus and nearby non-coding regions associated with HDL-C, we found that two SNVs, rs72786786 (chr16: 56985514 G/A; $p=2.52\times {10}^{-253}$) and rs12720926 (chr16: 56998918 A/G; $p=1.89\times {10}^{-260}$), both in moderate to high LD with the leading SNV rs17231506 (Figure S6), possess a MACIE-regulatory score greater than 0.99; rs72786786 has functional evidence, including binding proteins, motif changes, and in ENCODE DNase and regulatory chromatin states associated with diseases.⁴¹ Therefore, these two SNVs might be more functionally important than the leading SNV rs17231506 and might be prioritized for functional follow-up. For TG, there is also a lack of functional evidence for the leading SNV rs964184 (chr11: 116648917 G/C; $p=1.74\times {10}^{-157}$) in the APOA1/C3/A4/A5 gene cluster region. However, SNV rs2075290 (chr11: 116653296 C/T; $p=2.13\times {10}^{-103}$), which is in moderate LD with rs964184 at this locus, has a MACIE-regulatory score of 0.88 (Figure S7). These examples illustrate how MACIE scores can complement previous literature and provide additional information to aid prioritization of putatively functional causal variants for follow-up.

Figure 6.

LocusZoom plot for GWAS associations of LDL-C at the APOE locus

The lipid GWAS summary statistics were from the European Network for Genetic and Genomic Epidemiology (ENGAGE) Consortium (n = 58,381).⁴⁰ The MACIE-protein and MACIE-conserved scores for rs7412 are 0.96 and 0.97, respectively. The MACIE-conserved and MACIE-regulatory scores for rs1065853 are <0.01 and >0.99, respectively. LDL-C, low-density lipoprotein cholesterol.

Discussion

As the amount of publicly available annotation data increases and our understanding of variant functional effects continues to grow, describing variant functionality with a flexible yet practically interpretable and intuitive vocabulary will only become more important. Existing one-dimensional integrative scores cannot capture the multi-faceted functional profile of a variant because such ratings necessarily combine diverse, and possibly unrelated, sets of annotations into a single outcome. Oftentimes, they also ignore or do not fully consider the correlation between individual annotations. Current supervised methods further demonstrate performance profiles that are linked strongly to the quality of training-set labels. These supervised scores might lack robustness in the absence of gold-standard training sets.

In this paper, we have proposed MACIE, an unsupervised multivariate mixed-model framework that allows for multiple, possibly correlated, binary functional statuses. This framework offers several fundamental advancements over existing methods. First, MACIE provides multi-dimensional scores that measure functionality across multiple different functional classes. As posterior predictive probabilities, these scores are interpretable and scientifically relevant. They can be further summarized into marginal measures such as “probability of function according to at least one class of annotations” (MACIE-anyclass). The genome-wide average of the MACIE-anyclass score is 0.084, which is consistent with the prediction from previous studies.^20,42

Second, the MACIE model accommodates correlations both within and between classes. It has been reported that, although some of the available annotations measure similar notions of functionality, others provide distinct and complementary information.^9,22 By flexibly modeling potential, complex correlations across all the annotations, MACIE reflects this underlying biology. In doing so, it is better able to assign each annotation and group of annotations the appropriate amount of influence.

We showed that MACIE delivered powerful and robust performance in discriminating between functional and non-functional variants in multiple independent testing sets, including (1) validated, clinically relevant variants in ClinVar, (2) SGE-verified loss-of-function variants in BRCA1, and (3) experimentally validated functional variants in FANTOM5 and MPRA. Using lipid GWAS summary statistics from the ENGAGE Consortium, we illustrated that MACIE offers an effective tool for fine-mapping studies to prioritize putative functional variants for experimental follow-up. MACIE is also informative in prioritizing lipid-associated variants through a stratified LD-score regression-heritability enrichment analysis (supplemental materials and methods).^23,43 MACIE scores have already been used, for example, in the identification and characterization of inflammation and immune-related risk variants in squamous cell lung cancer.⁴⁴ Finally, the proposed MACIE scores can be used as a weighting scheme to further empower variant-set analyses of rare variants.⁴⁵

Our proposed MACIE framework provides a multi-dimensional functional-class extension of several existing unsupervised single scoring frameworks, such as EIGEN.¹⁵ MACIE fits a mixed model to the set of annotations for several latent functional classes and outputs the corresponding posterior component probabilities, which are highly interpretable. If we assume that there exists a single latent dichotomous variable summarizing functional status and that all annotations are independent on condition of the univariate functional status, then MACIE reduces to the GenoCanyon framework (Figure S1B).¹⁶ MACIE produces multi-dimensional functional scores and has advantages over GenoCanyon, which predicts the regulatory potential of genomic locations by using a single-dimensional score. The MACIE-damaging protein function score (AUC = 0.93) substantially outperformed GenoCanyon (AUC = 0.69) in the ClinVar coding testing set, and the MACIE-conserved score (AUC = 0.95) substantially outperformed GenoCanyon (AUC = 0.78) in the ClinVar non-coding testing set. In addition, the performance of the MACIE-regulatory score (AUC = 0.76) is comparable to that of GenoCanyon (AUC = 0.73) in predicting regulatory function, for example, in the dsQTL testing set.

The versatility of the MACIE approach does introduce additional decisions that investigators need to make. First, researchers need to decide which set of annotations to include and how to assign the annotations to different classes; both the quality of the annotations as well as the accuracy of their assignment can impact MACIE’s performance. In this paper, we assigned these annotation scores on the basis of biological knowledge. Second, the exponential family assumption in the model might also require a proper transformation of some individual annotation scores before model fitting. Third, users need to consider the trade-offs between a more complex model and computation time; these trade-offs include the number of classes, the number of functional scores in each class, and the number of variants used for training. Such issues will become more relevant when the MACIE framework is extended to integrate cell-type-specific, tissue-specific, species-specific, or phenotype-related annotations and to include more variants in the training set, for example from the Trans-Omics for Precision Medicine Program.^17,18,37,46 Nevertheless, these choices again highlight the flexibility of the MACIE approach. Unlike other one-dimensional algorithms that rely on assumptions more likely to be satisfied when the number of annotations is small, the MACIE statistical model scales well with increasing annotation data. Thus, MACIE can be expected to provide more meaningful predictions as the availability of variants and annotation scores continues to expand and the quality of these data improves.

A final important consideration in practical analysis concerns the differences between supervised and unsupervised methods. The performance of unsupervised scores might lag behind that of supervised methods when training datasets with relevant, high-quality labels are available. We observed this behavior when comparing MACIE to FATHMM-XF for ClinVar non-coding variants. Future extensions of interest include development of tools capable of integrating both supervised and unsupervised methods to further improve prediction accuracy.³⁷

Published: February 24, 2022

Data and code availability

The code for training MACIE models and MACIE scores used in all benchmarking examples are available for download at https://github.com/xihaoli/MACIE. Precomputed MACIE scores for all possible variants in the human genome are available for download at https://zenodo.org/record/5755656; https://zenodo.org/record/5756449; https://zenodo.org/record/5756479; and https://zenodo.org/record/5756563 (DOIs: 10.5281/zenodo.5755656; 10.5281/zenodo.5756449; 10.5281/zenodo.5756479; and 10.5281/zenodo.5756563). All genomic coordinates are given in NCBI Build 37/UCSC hg19.

Web resources

• 1000 Genomes, http://www.1000genomes.org

• BRCA1 SGE, https://sge.gs.washington.edu/BRCA1

• CADD, http://cadd.gs.washington.edu

• ChromHMM, http://compbio.mit.edu/ChromHMM

• ClinVar, https://www.ncbi.nlm.nih.gov/clinvar

• DANN, https://cbcl.ics.uci.edu/public_data/DANN

• EIGEN, http://www.columbia.edu/∼ii2135/eigen.html

• ENGAGE Consortium, http://diagram-consortium.org/2015_ENGAGE_1KG

• FANTOM5 CAGE, https://fantom.gsc.riken.jp/5/data

• FATHMM-XF, http://fathmm.biocompute.org.uk/fathmm-xf

• fitCons, http://compgen.cshl.edu/fitCons

• GENCODE, https://www.gencodegenes.org

• LINSIGHT, https://github.com/CshlSiepelLab/LINSIGHT

• LocusZoom, http://locuszoom.org

• MACIE, https://github.com/xihaoli/MACIE; https://zenodo.org/record/5755656; https://zenodo.org/record/5756449; https://zenodo.org/record/5756479; https://zenodo.org/record/5756563

• OMIM, http://www.omim.org