Mutation effects predicted from sequence co-variation

Abstract

Increasing interest in determining the effects of genetic variation for bioengineering, human health and basic biological research has propelled the development of technologies for high-throughput mutagenesis and selection. However, since designing functional assays is challenging and systematic testing of combinations of mutations is intractable, there is a parallel need to develop more accurate computational predictions.. Most computational methods have relied significantly on the signal of evolutionary conservation, but do not account for dependencies between positions in a sequence. We present an unsupervised method for predicting the effects of mutations (EVmutation) that explicitly captures residue dependencies between positions. We find that it improves the prediction accuracies of a comprehensive collection of recent high-throughput experimental fitness landscapes, biochemical measurements and human disease mutations. We suggest EVmutation may be useful to assess the quantitative effects of mutations in genes of any organism and provide precomputed predictions for ~ 7000 human proteins.

Introduction

Understanding the phenotypic effects of genetic variation is a central challenge for bioengineering and basic biology. Molecular technologists strive to engineer biologics that are safe and effective¹, design new genomes^{2, 3}, and develop ‘smart’ libraries of synthesized molecules^{4, 5}. Biologists seek to pinpoint the genetic changes that underlie organism phenotypes or complex diseases⁶ from the ever-expanding catalog of variation in humans and model organisms. Propelled by these needs new technologies have emerged that simultaneously assess the effects of thousands of mutations in parallel^{4, 7–27} (sometimes referred to as “deep mutational scans”²⁸ or ‘MAVEs’²⁹ ). In these assays, the measured attributes or processes vary, ranging from ligand binding, splicing and catalysis^{7, 11, 14, 16, 24, 30,26} to cellular or organismal fitness under selection pressure ^{8–10, 12, 15, 17, 20, 22, 23}.

However, although high-throughput mutational scans have improved our understanding of the consequences of genetic variation, their relevance to organism fitness and physiology critically depends on the choice of the measured phenotype^{29, 31}. It is reasonable that the relationship between a biochemical phenotype and organism fitness may be non-linear³² and even non-monotonic³¹. For the foreseeable future, scans are also limited in their scalability which limits researchers to focus either on a modest number of positions or alleles. Nature has not been subject to these limitations however, and proteins diverged by hundreds of positions can often functionally replace one another³³.

Statistical models of natural sequence variation can complement experimental approaches. These models take advantage of the fact that evolution has been performing its own massively parallel mutagenesis and selection experiments over time. Most computational methods including SIFT, PolyPhen-2, and CADD exploit evolutionary conservation to predict the effects of mutations^34–36 but their sequence analysis does not explicitly consider genetic interactions between mutations and the sequence background, despite widespread evidence for non-independence of the effects of mutations, known as epistasis^{37, 38}.

Recent work has applied unsupervised statistical models based on the co-evolution of natural sequences to accurately predict 3D contacts in protein and RNA structures^39–45, suggesting that dependencies between residues can be systematically captured from natural sequences. Although related models have been applied to predict the effect of mutations on a few proteins^46–48, these methods have not been tested systematically across many proteins or against human disease mutations.

We present a method, EVmutation, which accounts for epistasis by explicitly modeling interactions between all the pairs of residues in proteins (and bases in RNAs) and which can then quantify the effects of mutations, including multiple mutations, simultaneously. We compare predictions generated by our models with measurements from 34 mutagenesis experiments and in classification of human disease mutations showing that the modeled mutational landscapes are in better agreement with experimental data than existing non-epistatic methods.

Results

Probabilistic model captures residue dependencies

Extant proteins and RNAs exhibit strong signatures of selection throughout their evolution. It is not uncommon for genes separated by hundreds of millions of years, exhibiting negligible sequence identity, to nevertheless exhibit remarkable conservation of their structures and functions. Here, we aimed to build statistical models that can recover some of the dominant constraints that define families of homologous sequences.

We model the evolutionary process that has produced each family as a sequence generator at equilibrium that produces a sequence σ with probability P(σ) as

$$P(\sigma )=\frac{1}{Z}exp\left\{E(\sigma )\right\}$$ Eq. 1

Different parametric forms of the “energy” function E(σ) enable the model to capture different types of constraints on the sequences (epistatic or not) and Z normalizes the distribution to sum to one over all possible sequences of a fixed length. E(σ) may be thought of as a (negative) energy of a model from statistical physics or as proportional to the scaled fitness N_eF in toy, equilibrium models of population genetics⁴⁹. We use an energy function E(σ) with two types of constraints: pairwise constraints that describe co-dependencies in combinations of amino acids or nucleotides for each pair of sites, and site-specific constraints reflecting bias towards or away from specific amino acids or nucleotides at each position. The total energy for a specific sequences E(σ) is the sum of coupling terms J_ij between every pair of residues and a sum of site-wise bias terms h_i (fields),

$$E(\sigma )=\sum _i{\mathbf{h}}_i({\sigma }_i)+\sum _{i<j}{\mathbf{J}}_{ij}({\sigma }_i,{\sigma }_j)$$ Eq. 2

Combining the sequence generator model (Equation 1) with our energy function (Equation 2) produces a model that is known as a pairwise undirected graphical model in computer science and a Potts model in statistical physics. When fit to data, these models explain the global correlations observed between variables in a system in terms of direct pairwise interactions J_ij that are typically simpler and more localized. Determining these interactions involves simultaneous accounting for all possible couplings between all pairs of positions, which is not possible with local measures of correlation such as Mutual Information⁵⁰. When models of this form are fit to natural sequence families, the magnitudes of the J_ij terms have consistently predicted contacts in the 3D structures of proteins, with sufficient accuracy to predict the folds of proteins^{41, 43, 51, 52}.

We apply these models to make sequence-specific predictions about the relative favorability of mutations. Starting from a multiple alignment of a sequence family, we estimate the site and coupling parameters h and J using regularized maximum pseudolikelihood ^{40, 53–56}. After the parameters are inferred, we quantify the effects of single or higher-order substitutions on a particular sequence background with the log-odds ratio of sequence probabilities between the wild-type and mutant sequences (Fig. 1, Supplementary Fig. 1):

$$\mathrm{\Delta }E({\mathbf{\sigma }}^{\text{mut}},{\mathbf{\sigma }}^{\text{wt}})=log\frac{P({\mathbf{\sigma }}^{\text{mut}})}{P({\mathbf{\sigma }}^{\text{wt}})}=E({\mathbf{\sigma }}^{\text{mut}})-E({\mathbf{\sigma }}^{\text{wt}})$$ Eq. 3

Figure 1. Inferring context-dependent effects of mutations from sequences.

Evolution has generated diverse families of proteins and RNAs with varied sequences that perform a common function. An unsupervised probabilistic model trained to generate the natural diversity in a multiple sequence alignment of a family can be used to predict the relative favorability of unseen mutations. Left: Existing models describe functional constraints on each position i in a sequence σ independently, averaging over the effect of background positions j. This can lead to incorrect predictions of neutrality. Right: Our approach infers a global probability model with pairwise interactions between positions i and j (J_ij, see Methods) as well as background biases at single positions (h_i). For a more detailed graphical schematic of the calculation, see Supplementary Fig. 1.

The summation over coupling terms J_ij between all pairs of positions in the evolutionary statistical energy E directly incorporates sequence context, i.e. the effects of pairwise epistasis, into the calculation of mutation effects. We refer to this model, which is the basis of EVmutation, as the “epistatic model”.

Model captures experimental fitness landscapes

We assessed the extent to which the statistical energy landscapes of computed using EVmutation corresponded with experimentally measured changes of phenotypes. We collected data from both saturation mutagenesis experiments of proteins and RNA genes as well as focused low-throughput studies^{15, 46, 57–60} resulting in 34 datasets from 29 non-redundant experiments (21 proteins, and a tRNA gene²⁷; Supplementary Table 1) that includes all currently available mutation experiments where the protein or RNA had a sufficiently large and diverse alignment (Online Methods). For each protein or RNA tested, we generated a multiple sequence alignment of the sequence family, inferred the parameters h and J of the epistatic model, and compared the change in statistical energy (ΔE) to the corresponding experimental measurements of the effects of the mutations (Fig. 2; Supplementary Table 2). Since relationships between protein function and organismal fitness are not expected to be linear³¹, we focused on reporting rank correlations as the primary metric for evaluating predictive performance, but our results are robust to a variety of measures (Supplementary Table 3).

Figure 2. Saturation mutagenesis experiments provide a quantitative test of context-dependent predictions.

The computed ΔE mutational landscape of the DNA methyltransferase M.HaeIII (left, colour range from 5^th percentile to 0) agrees quantitatively with experimental measurements of M.HaeIII fitness under selection for cleaving activity of a restriction enzyme (right, ρ=0.69, N=1634; marginal distributions in orange). The average mutational sensitivity per position shows improved correlation beyond individual effects (ρ=0.80, N=304).

We found significant correlation between the computed ΔE of the epistatic model and the experimental measurements of phenotype and fitness for all high-throughput experimental datasets (Spearman’s ρ 0.4–0.7; p-values from <10⁻³⁰⁰ to <10⁻²⁷ and ρ=0.2, p<10⁻¹² for one of the BRCA1 experiments not expected to correlate well (see below)). We found that the agreement between ΔE and experimental data is higher if the assayed phenotype is closely linked to an essential process, and that agreement depends on the strength of purifying selection applied in the experiment (Fig. 3; Supplementary Table 3, Supplementary Fig. 2). For instance, some of the strongest correlations between ΔE and experimental data were for the enzymatic activity of a methyltransferase that protects DNA from degradation¹⁷, and of a β-glucosidase that hydrolyzes biomass⁴. Likewise, ΔE was more weakly correlated with data from experiments where the laboratory selection pressures may not match those in the natural environment and here the distribution of experimental mutation effects tends to be skewed towards mostly neutral effects (for example BRCA1 binding to BARD1) or towards mostly very deleterious (a bacterial kinase subject to high doses of kanamycin) (Supplementary Table 4, Supplementary Fig. 3). The dependence on the strength of selection is nicely exemplified by the increasing correlation of our predictions with the experiments on β-lactamase and kanamycin kinase as the antibiotic doses are titrated to reveal the full dynamic range of effects (Supplementary Figs. 4 and 5).

Figure 3. ΔE captures experimental fitness landscapes and identifies deleterious human variants.

(a) Computed effects of specific mutations (difference in evolutionary statistical energy ΔE) based on the epistatic model agree with diverse experimental measurements of fitness and molecular function for 34 experiments for 20 proteins, a protein complex and an RNA molecule (underlined) as measured by Spearman’s rank correlation coefficient ρ (for equivalent site average plot, see Supplementary Figure 2; for correlations across all different assays tested in the experiments, see Supplementary Fig. 4). (b) Evolutionary statistical energies ΔE distinguish human disease-associated variants from common alleles in the population. This separation increases with the minimum allele frequency (AF) of the variants assumed to be neutral (area under the ROC curve (AUC)=0.92 for AF≥0.1, AUC=0.94 for AF≥0.25, AUC=0.96 for AF≥0.5). (c) The epistatic model shows stronger agreement with experiments than the established methods SIFT and PolyPhen-2, a baseline model based on the BLOSUM62 substitution matrix, and a corresponding independent model without pairwise interactions (differences of ρ of more than 0.6 were included in the bin at 0.6).

The epistatic model also successfully captured the effects of mutations in low-throughput experiments on thermostability of Trypsin⁵⁸ (ρ=0.77, N=23) and SH3⁵⁷ (ρ=0.69, N=48) and the catalytic efficiency of β-lactamase¹⁵ (ρ=0.87, N=30), including double and triple substitutions. Although these experiments consist of a biased choice of mutants that may benefit our correlation, in principle, the results suggest EVmutation could be used to design stabilizing or catalytically stronger proteins.

To assess whether ΔE distinguishes human variants associated with diseases from putatively neutral variants, we compared predictions for 9008 variants (1553 proteins) annotated as pathogenic in ClinVar⁶¹, to variants in the 60,000 human exome sequencing collection (ExAC⁶) at increasing allele frequencies (AF; 2190 proteins; Online Methods). The ΔE scores for the disease variants are significantly different to those common alleles presumed mostly to be neutral from the ExAC collection (area under the ROC curve AUC=0.92–0.96, p-values <10⁻³⁰⁰, 2-sided sample KS tests; Fig. 3b).

Comparison of epistatic model with published methods

We sought to evaluate the utility of adding epistatic interactions to mutation prediction by comparing with commonly applied tools on both the assembled experimental data as well as human genetic variants. For the experimental data, we compared predictions from EVmutation with SIFT³⁴ and PolyPhen-2⁶², along with the substitution matrix BLOSUM62. The rank correlations of ΔE (epistatic model) are higher than the other methods for the majority of the analyzed mutation experiments (Fig. 3c; Supplementary Table 5). The average increase in ρ is 0.30 over the BLOSUM62 matrix (EVmutation better on 30/31 datasets) and 0.18 over SIFT (better on 29/31 datasets) and 0.18 over PolyPhen-2 (better on 26/29 datasets). Whilst this work was in review, another study used an epistatic model to predict effects of mutations in β-lactamase ⁴⁸, but using different statistical inference and alignment generation protocols. EVmutation predictions are mildly more accurate when compared directly across the four proteins tested (average increase in correlation: 0.04) and on a par with a computationally expensive approach for inference for a small number of HIV protein mutations⁴⁶ (EVmutation ρ=0.81 vs. ρ=0.80; Supplementary Table 6).

We compared predictions from the unsupervised EVmutation method with PolyPhen-2, a supervised mutation predictor, on human disease variants. Despite no training on clinical variants, the performance of ΔE predictions are comparable with those of PolyPhen-2 and SIFT on the HumVar benchmark set^62–64 (ΔE: AUC=0.89; PolyPhen-2: AUC=0.88; SIFT: AUC=0.85) and better than PolyPhen-2 with the subset of variants that are predicted differently by PolyPhen-2 and SIFT (AUC 0.71 versus AUC 0.61; Supplementary Fig. 6). It has been reported that prediction methods, such as PolyPhen-2, that use supervised machine learning, can have inflated measures of accuracy due to multiple routes of circularity in the training and test sets^{16, 64}. Therefore it is surprising that our unsupervised model performs as well, or better than, supervised methods that have been directly trained to segregate deleterious human variants.

Interactions terms underlie improved predictions

In order to understand whether the improved predictions of our epistatic model result from the added pairwise interactions J, we used a control, non-epistatic, “independent” model that is identical to ΔE but lacks interactions between sites. We fit this model to the same alignments as the epistatic model (Methods). The independent model is closely related to the log-frequency, Position-Weight Matrices and other scores of conservation ^{16, 34, 36, 65} that are commonly used by existing methods. The ability of ΔE (epistatic model) to capture mutational effects is better (difference in ρ ≥ 0.05) than the independent model across 23 of 34 of the analyzed experiments (Fig. 3c; Supplementary Table 5; average increase of ρ: 0.14, including high-throughput fitness experiments with well-defined selective pressures such as β-lactamase^{10, 15, 18, 48}, kanamycin kinase¹³ and DNA methyltransferase M.HaeIII)¹⁷ and is also more accurate for human disease mutations (AUC 0.92 versus 0.88 for AF≥0.1), despite the ascertainment bias of these mutations⁶⁴.

One particularly striking contrast between the epistatic and independent models is for the toxin-antitoxin complex ParED²⁵, especially when the two proteins are modeled jointly as a protein interaction ⁴⁰. In this case the epistatic model captured the experimental effects with much higher correlation (ρ=0.51 for singles, doubles and triples, N=3335 and ρ=0.43, N=9194 for all mutations) than the independent model, which hardly correlated at all (ρ=−0.05 for all up to triplets). Interestingly, if ΔE is computed on couplings from ParD alone, the match to experiment is lower (ρ=0.33) and when computed only on couplings between (but not within) the complex subunits, the predictions are comparable to the full model (ρ=0.53 vs ρ=0.51). Hence the interactions learned in the epistatic model support accurate prediction of the interaction specificity of the complex²⁵.

Experiments where the epistatic model performs comparably to the independent model (11/34 sets) tend to be where the correlations are below average for either method (Supplementary Table 5). This includes three of the four viral proteins and may be a consequence of the limited diversity of the sequence alignments or alternatively, a discrepancy between the proxy for viral fitness in the laboratory and the in vivo fitness of the virus²².

We next asked how the results for the epistatic and independent models depend on the evolutionary depth of the alignments used for inference. As sequence alignments become narrower around the mutated protein, i.e. evolutionarily distant sequences are progressively excluded, the epistatic model and independent model make similar predictions that both perform less well against experimental data. For specific alignment depths of some proteins, the independent model can capture the experimental data as accurately as the epistatic model (Supplementary Table 7). However, since we do not know which alignment depth to choose a priori, the ability of the epistatic model to capture constraints without prior knowledge of the optimal alignment depth may be an advantage. The choice of alignment in our method is blind to the mutation experiments and is based on the algorithm used for alignment choice when computing 3D structure contacts using EVfold⁴¹ (Methods).

Where is the epistatic model better?

To investigate where inclusion of epistatic interactions leads to improved predictions, we compared the epistatic and independent models on a mutation-by-mutation basis. Direct comparison of residuals can be misleading when the functional relationship between predictions and experimental measurements is unknown and/or nonlinear, so we computed deviations between the epistatic and independent models after mapping through a quantile-quantile transformation. This allowed us to identify specific mutations that have the largest reduction of prediction error when using the epistatic model. For instance, in an RNA-recognition motif of the poly(A)-binding protein, the 3 mutations with the largest reduction in error using the epistatic model are H172T, N127R and S154D. The epistatic model agrees with experimental data showing deleterious effects of these mutations, but the independent model suggests that the same mutations are neutral or even beneficial (Fig. 4a, Supplementary Table 8). Although these substitutions are frequently observed in other sequences in the protein family alignment, only the epistatic model captures that they are not acceptable in the background of the target sequence. These three residues form a network of spatially proximal residues that contact RNA (Fig. 4a), and are highly coupled when their total coupling is summarized across all amino acid combinations (Supplementary Table 9). The predictions that differ most between the epistatic and independent models (defined as >2σ of error change distribution) are in 8 positions that are within 6 Å of the bound RNA, which includes the positions identified above (Fig. 4b).

Figure 4. Improvements of the epistatic model for functional sites.

(a) Left: The RNA-binding residue H172 of PolyA-binding protein (PABP) is strongly coupled to other residues in the binding interface that are close in 3D. Right: The epistatic coupling leads to strong constraints on acceptable amino acids in position 172, as observed in an experimental mutation scan of PABP. Only the epistatic model correctly identifies these co-constraints, while a model without sequence context (independent model) suggests many more substitutions would be acceptable (range of experimental preferences scaled to range of predicted preferences based on full set of mutants for entire domain). (b) Positions in PABP for which prediction accuracy improves the most by considering epistasis (≥2σ difference in root mean squared prediction error, spheres) cluster around the RNA ligand (yellow sticks, PDB: 4f02). (c) For seven high-throughput datasets where the correlation ρ of the epistatic and independent models differs more than 0.05, the epistatic model is more accurate overall (1^st column), specifically for the effects of mutations of residues in interaction and ligand-binding sites (2^nd column), where the residue mutation is rare versus frequent in the evolutionary sequence alignment (3^rd and 4^th columns, and where the residue change is damaging versus neutral in the experiment (5^th and 6^th columns) (Methods and Supplementary Table 8).

We then explored whether these observations generalize to other proteins by applying the above analysis for all proteins with single substitution high-throughput scans in which the epistatic and independent models differed (ρ(ΔE(epistatic)) − ρ(ΔE(independent)) > 0.05) but still showed reasonable correlation to the experimental data (ρ>0.50; Fig. 4c, Supplementary Fig. 7). Comparison of predictions made by the two models suggests that the advantages of the epistatic model stem at least in part from more accurate modeling of positions that facilitate interactions with ligands or other proteins, and show strong deleterious effects in experiments (Fig. 4c; Supplementary Fig. 7).

Discussion

We report here that a prediction method built on natural sequence variation can partially capture the experimental effects of mutations in a variety of biological contexts. This can be applied to predict or interpret the effects of genetic variation in any species of interest, and for higher-order mutations that change multiple positions at the same time.

Limitations of the model include biases that arise from evolutionarily-younger families of limited diversity, non-uniform selective constraints across a sequence family and higher order epistasis. Although incorporating epistatic terms into the model results in a practical improvement over other methods, there are remaining challenges regarding the interpretation and inference of the model parameters. For interpretation, it is difficult to distinguish those couplings in the model that may be due to subfamily-specific selection from those that may be due to more universal epistatic constraints across the whole family⁶⁶ and future methods may begin to address these issues by consideration of phylogeny. For inference of the parameters, the statistical challenge remains that the typical number of free parameters (~10⁶–10⁸) vastly exceeds the number of available sequences (~10³–10⁵) and since evolutionary sampling is highly correlated and consequently highly redundant, this may lead to significant challenges in extrapolating a reasonable space of possible sequences.

The success of this and other models based on sequence variation at recapitulating high-throughput mutation experiments depends in part on the extent to which experimental assays capture phenotypes that are under direct, long-term selection (Fig. 5). For example, thermostability, activity, or binding energetics of a protein will generally not all contribute to fitness in the same way, so even a perfect model and perfect measurements might have imperfect correlations. The excellent correlation between model and experimental data for β-lactamase plausibly reflects how survival ‘in the wild’ in the presence of β-lactam antibiotics depends directly on that specific protein. For other assays, such as nonessential peripheral enzymes or signaling proteins, the property being tested in the lab may have only indirect, context-dependent impact on the organism. So the evaluation of agreement between model and experiment is two-way, reliant on whether the model is effective as a method and, when multiple kinds of measurements are available for the same protein, to what extent the lab assay captures evolutionarily conserved properties.

Figure 5. Computational predictions complement experimental measurements.

Various molecular phenotypes (center) such as structure, thermostability, activity, and ligand-binding affinity are determined by genotype and contribute to fitness in a complicated manner that is not known a priori. However, the distribution of contemporary genotypes (left) provides a record of historical fitness values (right) which can roughly be inferred by computational methods. Identifying those phenotypes that connect to inferred fitness may shed light on which molecular phenotypes have historically been the most relevant to the organism.

Nevertheless, our probabilistic approach is readily applicable for analysis of specific variants and combinations thereof for numerous families of proteins and RNAs and the interactions between them. We anticipate that analyses of genetic variation and mechanisms of evolution will benefit from global probability models of sequence families that explicitly incorporate interactions between positions, such as the model presented here. The consistency of our predictions with prior biological knowledge on functional sites highlights how inclusion of epistatic interactions will facilitate more accurate assessment of mutations and aid the design of libraries of protein sequences. To enable the community to analyze ΔE for their proteins of interest, we make software (Supplementary Code) and pre-computed mutation effects for ~ 7000 human proteins and available at evmutation.org.

ONLINE METHODS

Generation of multiple sequence alignments

For each analyzed protein (target sequence), multiple sequence alignments of the corresponding protein family were obtained by the default five search iterations of the profile HMM homology search tool jackhmmer⁶⁷ against the UniRef100 database of non-redundant protein sequences⁶⁸ (release 11/2015). To control for comparable evolutionary depth across different families, we used length-normalized bit scores to threshold sequence similarity rather than E-values⁴⁰. A default bit score of 0.5 bits/residue was used as a threshold for inclusion unless the alignment yielded < 80% coverage of the length of the target domain or if there were not enough sequences (redundancy-reduced number of sequences ≥10L); in the first case, the threshold was increased in steps of 0.05 bits/residue until sufficient coverage was obtained; in the second case, the threshold was decreased until there were sufficient sequences (≥10L). If these two objectives were conflicting, precedence was given to maintaining more than 10L sequences. Since the sequence diversity of viral protein families is typically much lower than that of bacterial and eukaryotic families, the alignment depth for viral proteins was chosen as a default of 0.5 bits/residue even if the redundancy-reduced number of sequences was lower than 10L. The alignments were post-processed to exclude positions with more than 30% gaps and to exclude sequence fragments that align to less than 50% of the length of the target sequence. For the ParE-ParD toxin-antitoxin interaction, a joint sequence alignment with matched homologs of both interaction partners was generated using our previously described approach EVcomplex⁴⁰. Alignments for RNA sequence families were obtained from the Rfam database⁶⁹ and redundancy-reduced at the same 80% identity cutoff as proteins. The tRNA alignment was filtered to contain only sequences with a CCU anticodon.

Inference of epistatic models of biological sequences

Model

Each family is modeled as a distribution over the space of all possible sequences that is parameterized by two types of constraints: site-specific constraints on the biases for specific amino acids or nucleotides at each position, and pairwise constraints for combinations of amino acids or nucleotides for each pair of sites. The use of such a global probabilistic approach is motivated by the idea that the induced correlations observed in a multiple sequence alignment can be explained by a simpler set of underlying couplings between positions. Local measures of coupling, such as the log ratios that define Mutual Information, cannot deconvolve these transitive correlations between positions⁵⁰.

The form of the chosen distribution can be thought of as the least-structured (i.e. maximum entropy) distribution over sequence space that is consistent with the single-site and pairwise marginal distributions of amino acids or nucleotides observed in the alignment^{43, 50, 55, 70}. Under the model, the probability of a sequence σ of length N is defined as

$$P(\sigma )=\frac{1}{Z}exp\left\{E(\sigma )\right\}$$

The partition function Z normalizes the distribution by summing over the relative weights (Boltzmann factors) of all possible sequences σ′. The strength of the Boltzmann factors for each sequence σ is defined by the evolutionary statistical energy E(σ) as the sum of all its pairwise coupling constraints J_ij and single-site constraints h_i (fields), where i and j are positions along the sequence:

$$E(\sigma )=\sum _i{\mathbf{h}}_i({\sigma }_i)+\sum _{i<j}{\mathbf{J}}_{ij}({\sigma }_i,{\sigma }_j)$$

In a Maximum likelihood fit without regularization, the site-specific parameters h_i(σ_i) and pair-specific parameters J_ij(σ_i,σ_j) implement the constraints that the marginal probability distributions of the model agree with the empirical marginal frequencies f_i(σ_i) and f_ij(σ_i, σ_j) in the sequence data. This model is also as a Markov Random field or a Potts model in statistical physics.

Insertions and deletions

While the model readily describes fixed-length sequences such as strings of 20 amino acids or 4 nucleotides, it is unclear how to meaningfully represent the insertions and deletions in a way that reasonably reflects the generative process occurring in evolution. Traditionally, statistical approaches have tended to model indels either as (i) an extra character or (ii) missing data. Both of these approaches are conceptually problematic, since the former partitions a single deletion event into many separate ‘gap characters’ while the latter forces the model to impute a hidden variable in gapped positions that can affect other variables in the system despite knowledge that no such coding variable exists. We used an alternative approach for modeling indels that does not invoke a gap character and does not give missing data the ability to influence the system. We regard the indel process as a separate, observed process and instead model the conditional distribution of the amino acids given an indel pattern. If zi are Bernoulli indicators that are 1 when a site is coding and 0 when gapped the conditional energy function for coding regions is:

$$E(\sigma ;\mathbf{z})=\sum _i{z}_i{\mathbf{h}}_i({\sigma }_i)+\sum _{i<j}{z}_i{z}_j{\mathbf{J}}_{ij}({\sigma }_i,{\sigma }_j)$$

Given an observed set of gaps, the marginal distribution of the amino acids at the ungapped positions defines a conditional model for a sequence. In principle, this approach could be made fully generative by combining it with a model for gaps, but for the purposes of this study we only fit the conditional distribution.

Sample reweighting

Natural sequences descend from a common ancestry and consequently may be highly correlated. Moreover, uneven sampling of the family by both sequencing projects and natural processes may result in over-abundance of sequences from some parts of phylogeny (e.g. model organisms) and an under-abundance of others. To partially account for this redundancy, we use the established approach of reweighting sequences by a measure of their uniqueness^{43, 44}. Briefly, we define the sequence weight π_s of a given sequence as ${\pi }_s={\left({\displaystyle \sum _t}I\left[D_H\left({\sigma }^s,{\sigma }^t\right)<\theta \right]\right)}^{-1}$ where D_H(σ^s,σ^t) is the normalized Hamming distance (i.e. 1.0 - percentage identity) between sequences s and t and θ is a threshold for the percent divergence of sequences classified as similar. We used an 80% identity cutoff (θ = 0.2 ) for all proteins except viral proteins, for which we used a 99% identity cutoff (θ = 0.01).

Inference

Given observed sequence data (with potential reweighting for redundancy), the model parameters h_i and J_ij could in principle be estimated by maximum likelihood, i.e. by finding the parameters that maximize the probability of observing the sequence data. This approach, however, is deterministically intractable due to the 4^N (RNA) or 20^N (protein) sequences in sequence space that must be to compute the partition function Z. As a replacement for the full likelihood, we used a site-factored pseudolikelihood approximation⁵³.

Regularization

The number of parameters for the pairwise model, which for typical protein families of 50–500 amino acids will range from 10⁵ to 10⁷, outnumbers the typical number of sequences available (10² to 10⁵) for even the largest families by several orders of magnitude. In this under-sampled regime, standard maximum likelihood estimation is highly prone to overfit the sample data. We penalized model complexity with l₂-regularization^{55, 56}, which may also be interpreted as maximum a posteriori (MAP) inference under zero-mean Gaussian priors on the parameters. The strength of l₂-regularization was set as λ_h=0.01 for the single-site constraints h_i and λ_J=0.01·q(N−1) for the pairwise coupling constraints J_ij, where N is the number of sites in the model and q corresponds to the number of possible states (q=20 for proteins, q=4 for RNA). Combining the pseudolikelihood approximation, sequence reweighting, treatment of indels, and regularization, we estimate the parameters by maximizing the objective

$$\left\{\mathbf{h},\mathbf{J}\right\}=\underset{\mathbf{h},\mathbf{J}}{argmax}\sum _{s,i}{\pi }_{s}log{P}_i^s(\sigma )-{\lambda }_{\mathbf{h}}\sum _{i,a}{\mathbf{h}}_i{(a)}^2-{\lambda }_{\mathbf{J}}\sum _{i,j,a,b}{\mathbf{J}}_{ij}{(a,b)}^2$$

where the log conditional likelihood $P_i^s(\sigma )$ is defined whenever position i in sequence s is ungapped as

$$P_i^s(\sigma )=exp\left({\mathbf{h}}_i({\sigma }_i^s)+\sum _{j\ne i}{z}_j^s{\mathbf{J}}_{ij}({\sigma }_i^s,{\sigma }_j^s))\right)/\sum _{a}exp\left({\mathbf{h}}_i(a)+\sum _{j\ne i}{z}_j^s{\mathbf{J}}_{ij}(a,{\sigma }_j^s))\right)$$

and is 0 otherwise. We solve this optimization problem with a quasi-Newton method (L-BFGS), and make our C software implementing this available at github.com/debbiemarkslab/plmc.

Calculation of context-dependent mutation effects (evolutionary statistical energy)

Using the inferred probability models, one can then quantify the effect of single or higher-order substitution on a particular sequence background by computing the log-odds ratio of probabilities between the mutant sequence σ^mut and the wild-type sequence σ^wt. This ratio of probabilities is simply the difference of the evolutionary statistical energies of the two sequences, which is

$$\mathrm{\Delta }E({\mathbf{\sigma }}^{\text{mut}},{\mathbf{\sigma }}^{\text{wt}})=\sum _i\left({\mathbf{h}}_i({\sigma }_i^{\text{mut}})-{\mathbf{h}}_i({\sigma }_i^{\text{wt}})\right)+\sum _{i<j}\left({\mathbf{J}}_{ij}({\sigma }_i^{\text{mut}},{\sigma }_i^{\text{mut}})-{\mathbf{J}}_{ij}({\sigma }_i^{\text{wt}},{\sigma }_i^{\text{wt}})\right).$$

For a given mutant, the evolutionary statistical energy difference ΔE will be the sum of differences of the single-site constraints h_i for all substituted sites, plus the sum of differences of the coupling parameters for all pairs of positions involving at least one mutated site. By evaluating the change of couplings to other sites, the sequence context and therefore epistatic effects are explicitly incorporated into the computed mutation effects. Values of ΔE above 0 correspond to more probable mutant sequences (putatively beneficial), values below 0 to less probable mutant sequences (putatively deleterious) and values equal to 0 to equally probable sequences (putatively neutral).

Throughout the manuscript, mutation effects computed using this model are referred to as statistical energy differences from the epistatic model.

Calculation of evolutionary couplings

Between any two positions i and j in the protein family, the coupling matrix J_ij describes the co-constraint on all possible 20² amino acid or 4² nucleotide combinations. To quantify the total epistatic constraint between pairs of sites across all sequences in the alignment, the Frobenius norm was used to summarize each matrix J_ij into a single number that is proportional to the standard deviation of the (zero-mean) matrix⁵⁶. After computing the norm scores for every pair of sites, we remove background coupling that is presumed to be caused by limited sampling and phylogenetic relationships between sequences by applying the average product correction (APC)⁷¹. This is equivalent to setting the dominant singular value or eigenvalue of the coupling matrix to zero. Significantly constrained pairs (evolutionary couplings, ECs) were then selected by a mixture model-based strategy quantifying the probability of any pair to belong to the high-scoring tail of the score distribution rather than the background noise distribution^{40, 72}.

Inference of independent statistical models

To assess the contribution of epistatic interactions to ΔE, additional maximum entropy models were inferred that describe protein sequences using only site-specific amino acid constraints h_i, without considering explicit inter-dependencies between sites (i.e., predict mutation effects independently of the sequence context). Since the parameters of each independent model were inferred from the same sequence alignment as the epistatic model, it serves as a direct control for the contribution of epistatic interactions. The probability of any amino acid sequence σ under this “independent” model is given by

$$P(\sigma )=\frac{1}{Z}exp\left\{\sum _i{\mathbf{h}}_i({\sigma }_i)\right\}$$

Consistent with the regularization applied to the epistatic model, the strength of the l₂ penalty was set to λ_h=0.01 when estimating the model parameters h_i. The statistical energy difference between two sequences can be analogously inferred by calculating the statistical energy difference between the mutant and the wild-type sequences, which again corresponds to the logs-odds ratio of their probabilities. This formalism is closely related to the log-odds conservation scores used in many methods to predict mutation effects from sequence^36,73.

Mutational landscape datasets

Mutation effect datasets were identified by a comprehensive literature search for quantitative high-throughput mutagenesis experiments of entire proteins, protein domains or RNA molecules. All experiments that targeted proteins or RNAs with insufficient sequence diversity (redundancy-reduced number of sequences <10L, where L=length of protein or domain), covered only small subregions, or tested synthetic wild-type proteins were excluded from the final compilation of datasets (Supplementary Table 1). For further comparisons, the dataset was extended with low-throughput measurements of molecular phenotypes (stability, catalytic activity, binding), with a focus on sequence co-evolution studies. If a dataset reported more than one measurement for the wild-type sequence (as was the case for 11 of the high-throughput scans with one wild-type value per position in the sequence), we averaged these redundant experimental measurements into a single value (Supplementary Table 2).

Classification of experimental mutation effects

18 of the experimental mutation scans had visible bimodality in their effect distribution. Since this can lead to biased Pearson and Spearman correlations, we tested these with a binary classification measure, Mathews Correlation Coefficient, in addition to the other analyses. We classified mutations as damaging or neutral by (i) fitting a two-component Gaussian mixture models to each data set, and (ii) by assigning individual mutations to the mixture model component returning the higher posterior probability. (Mutation effects (enrichment ratios of sequencing reads before and after functional selection) were transformed into log-space where the experiment was reported in linear space.)

Correspondence between computed and experimental mutational landscapes

The agreement between computational and experimental mutation effects was evaluated using standard metrics of bivariate dependence. Due to strongly ed experimental effect distributions and the expected non-linear relationships between protein function and organism fitness we focused on Spearman’s rank correlation coefficient with dense ranking as our main evaluation metric. To test the robustness of our results, we additionally evaluated all relationships using distance correlations⁷⁴ and, where applicable, Pearson correlation coefficients and the corresponding linear regression R² values. For datasets showing bimodal effect distributions, we also tested prediction quality in a binary setting using the Matthews correlation coefficient⁷⁵. Quantitative ΔE values from the epistatic and independent models were assigned as damaging or neutral if they were below or above the median of the respective effect distribution, respectively.

Analysis of properties of distributions

Systematic biases in experimental or computed effect distributions can influence which correlation measures are applicable, and what bivariate relationship between the two can be expected. To describe the overall shape of distributions, in particular deviations from normality and strong biases towards damaging or neutral effects, for each distribution we calculated its skewness (scipy.stats.skew) and the R² of the sample data against a normal distribution in a probability plot (scipy.stats.probplot). The latter quantity corresponds to the test statistic of the Shapiro-Francia test for normality.

Comparison to existing mutation effect prediction methods

For the comparison of ΔE to existing approaches for the prediction of mutation effects, we computed mutational landscapes using local installations of SIFT and PolyPhen-2 applied to the same input sequences as for our sequence alignments. Quantitative effect scores and binary classifications (neutral/damaging) were obtained from the respective columns in the output files. Mutation effects computed using the BLOSUM62 matrix correspond to the respective matrix entry of wild-type and substituted residue. Correlations were calculated on the joint set of variants that could be predicted by all methods.

Human gene and disease variant analysis

Besides the evaluation on quantitative mutation effect data, we assessed the ability of ΔE to discriminate between known disease mutations and neutral amino acid variants in a binary classification setting. For this evaluation, we computed quantitative effect scores and tested how well both types of variants are separated by ΔE. Unlike the existing method PolyPhen-2 which trains on known disease variants, our mutation effects are purely based on sequences without learning on the outcome variable.

Alignments for protein sequences with disease and/or neutral variants were generated by identifying Pfam domains in the respective sequence using hmmscan from HMMER and running our own alignment protocol for each domain region. If possible, regions were extended by 10 residues on either side to correct for the lack of N- and C-terminal coverage often observed in Pfam. Following the same protocol as for mutational scans, alignments were inferred at a bitscore threshold of 0.5 bits/residue. Families where the number of sequences in the alignment exceeded our computational resources were run at a bitscore of 0.8 bits/residue. Alignments were filtered at a 95% sequence identity cutoff to reduce computation time. Epistatic and independent models were then inferred for all alignments and used to compute the effect of amino acid substitutions. In the evaluation, we used all variants in domains that had alignments with M_eff/L ≥ 1 and could be unambiguously mapped to UniProt sequences.

To assess if ΔE separates disease and neutral variants, we derived a dataset based on high-confidence disease and neutral variants. We obtained 15405 amino acid variants in 2387 proteins from ClinVar⁶¹ that were unambiguously annotated with clinical significance “pathogenic”. Similarly, we derived sets of amino acid variants assumed to be neutral because of their high allele frequency in the ExAC exome sequence dataset⁶, at increasing levels of stringency (allele frequency (AF)≥0.1: 13643 variants/6993 proteins; AF≥0.25: 8595 variants/5193 proteins; AF≥0.5: 4700 variants/3282 proteins). Of the pathogenic ClinVar variants, 10556 were covered by an alignment (in 1848 proteins), and 9008 (1553 proteins) had sufficient sequences (M_eff/L ≥1) and were used for evaluation. Of the ExAC variants, 3514 variants (2190 proteins) remained at AF≥0.1, 2193 variants (1524 proteins) at AF≥0.25 and 1182 variants (937 proteins) at AF ≥0.5.

For comparison to the existing mutation effect classifiers SIFT and PolyPhen-2, we chose the HumVar dataset (as provided by Grimm et al.)^{62, 64} as this is commonly used as a benchmark. However, using this benchmark set in the comparison to our unsupervised method is conservative, since most current supervised methods, including PolyPhen-2, train in a supervised way on these known variants and may have a tendency to over estimate their own accuracy⁶⁴. Ideally, one would construct a truly unbiased test set by excluding these and related variants, but since this leads to a strong reduction in the number of variants that can be used for evaluation, we left the benchmark set as used by others⁶⁴. We started from 40389 variants (9231 proteins) in the dataset, and after using the above domain identification and alignment protocol, arrived at a set of 21915 variants in 5067 proteins jointly predicted by all methods. Of these, 18001 variants in 3912 proteins had M_eff/L≥1 and were used for evaluation. In addition to the full set of variants with sufficient alignment depth, we also evaluated performance on “difficult” examples where the predictions between SIFT and PolyPhen-2 disagreed (3126 variants in 1459 proteins).

The statistical energy distributions between neutral and disease variants were compared using two-sample Kolmogorov-Smirnov tests (two-sided; function scipy.stats.ks_2samp). To assess the discrimination both types of variants, the area under the receiver operating characteristic curve (AUC) was calculated using scikit-learn (function sklearn.metrics.roc_auc_score) and compared between the different methods. We also evaluated the discovery of damaging variants with high specificity by computing the partial AUC up to a false positive rate of 20% using the pROC R package⁷⁶.

Pre-computed mutation landscapes for human proteins

Since we calculated epistatic models for all human proteins that have variants annotated as pathogenic, ExAC variants with allele frequency ≥ 0.1, and others in the course of this analysis, this results in a resource of models for 6955 unique human proteins (9957 alignment regions, M_eff/L≥1). For these proteins, we provide single substitution landscapes, sequence alignments, and evolutionary couplings (summarized epistatic constraint for pairs of positions) at evmutation.org. Mutational landscapes can be explored using interactive visualization. Together with the provided software (Supplementary Code), the downloadable files allow to reproduce our analyses and compute higher-order mutation landscapes for any of these proteins.

Error analysis of evolutionary statistical energy predictions

Besides the overall correlation analysis between prediction and experiment, we wanted to understand how large the prediction error is for individual mutations, and if particular mutations were predicted with lower error by either the epistatic or the independent, independent model. To calculate individual error terms, we need to compare the predicted value to the respective experimental value. In our case, this is however complicated by the fact that ΔE and the experimental data are on different scales and the relationships between both are often non-linear, with no particular expectation on the shape of the relationship. This problem prevents the use of standard regression approaches to calculate error terms.

Instead, we chose to transform predicted values into the space of the experiment by a quantile mapping strategy (a perfect prediction would mean that the normalized ranks of each variant in the predicted and experimental distribution agree). For any substitution S, let ΔE(S) correspond to the p-quantile of the predicted distribution. Then we transform a prediction ΔE(S) into experimental space by mapping it onto the respective p-quantile of the experimental distribution. We denote this mapping by Q(ΔE(S)). For robustness against a small number of experimental outliers, Q assigns the respective experimental 0.01/0.99 experimental quantile for any value below/above the predicted 0.01/0.99 quantile. By defining a quantile-quantile mapping for each of the models, we can now calculate an error term ε(ΔE(S)) for each variant S by comparing the mapped prediction Q(ΔE(S)) to the actual experimental value y(S) of the variant S: ε(ΔE(S)) = Q(ΔE(S)) − y(S). The error term ε(ΔE(S)) is normalized by the range of the full experimental distribution between the 0.01 and 0.99 quantiles to make it comparable between different experiments.

To evaluate if the use of epistatic interactions improves or decreases the error of individual variant predictions (i.e., variants that are predicted with different log-odds scores ΔE(S) compared to the independent model rather than similar log-odds scores), we had to modify the above approach because differently predicted, but wrong variants might bias the quantile-quantile mappings in favor of each method. Instead, we defined quantile-quantile mappings Q′ not on the full distributions but only on those variants that are predicted similarly between both models (|ΔE(epistatic) − ΔE(independent)| ≤ 2, threshold chosen so that there are enough variants to define a smooth curve). Q′ allows to map any arbitrary score based on the nearest quantile that was used during calculation of the mapping. Comparison of the respective absolute error terms ε(ΔE(S)) = Q′(ΔE(S)) − y(S) between the epistatic and the independent model allows to assess for any substitution S if the use of epistatic interactions increases or decreases the error compared to the experiment.

The computation of error terms for individual variants makes it possible to assess if certain groups of variants are systematically predicted more or less accurately using epistatic interactions. For this analysis, we compared the root mean square errors (RMSE, summing the individual error terms ε(ΔE(S))) across all individual substitutions within a particular group between the epistatic and the independent model. Subgroups were defined as follows: (1) High/low experimental effect: Gaussian mixture model of effect distribution as described above; (2) Frequent/Rare substitution: frequency of substitution in respective column of alignment ≥ 0.01 or < 0.001; (3) Ligand binding/protein interaction: all substitutions to residues within 4Å minimum atom distance of ligand or interaction partner, but excluding evolutionarily conserved cofactors; (4) Buried/Exposed: Relative solvent accessibility calculated using DSSP < 0.1 or > 0.25; (5) Conserved/Variable: Column conservation of position in alignment > 0.4 or < 0.2; (6) Strongly/weakly coupled: At least 4/no coupled pairs in list of N top-ranking long-range (|i − j|>5) evolutionary couplings.

Analysis of structural features

Evolutionary couplings calculated from multiple sequence alignments were compared to experimental protein 3D structures from the PDB⁷⁷ to assess if the identified epistatic constraints correspond to structural contacts. Structures and mappings to the target sequence were obtained using jackhmmer-based searches against the PDB (one search iteration), and residue pair distances calculated for up to 10 of the most significant hits with a normalized bit-score of at least than 0.5 bits/residue to the target sequence. Two residues were considered to be in contact if any of their atoms are closer than 5 Å in any of the identified structures; a distance threshold of 4 Å was applied to interactions between amino acid residues and ligands.

Data analysis and method availability

All data analysis was conducted using Jupyter notebooks⁷⁸ and the scientific Python stack^{79, 80}. Supplemental Web Data, human protein predictions and code (Supplementary Code) to calculate statistical models and mutation effects from sequence alignments are available at evmutation.org.

Editors summary.

The global effects of epistasis on protein and RNA function are revealed by an unsupervised model of amino acid co-conservation in evolutionary sequence variation.