Gene-specific features enhance interpretation of mutational impact on acid alpha-glucosidase enzyme activity

Abstract

We present a computational model for predicting mutational impact on enzymatic activity of human acid alpha-glucosidase (GAA), an enzyme associated with Pompe disease. Using a model that combines features specific to GAA with other general evolutionary and physiochemical features, we made blind predictions of enzymatic activity relative to wildtype human GAA for >300 GAA mutants, as part of the CAGI 5 GAA challenge. We found that gene-specific features can improve the performance of existing impact prediction tools that mostly rely on general features for pathogenicity prediction. Majority of the poorly predicted mutants that lower wildtype GAA enzyme activity occurred on the surface of the GAA protein. We also found that gene-specific features were uncorrelated with existing methods and provided orthogonal information for interpreting the origin of pathogenicity, particular in variants that are poorly predicted by existing general methods. Specific variants in GAA when investigated in the context of its protein structure suggested gene-specific information like the disruption of local backbone torsional geometry and disruption of particular sidechain-sidechain hydrogen bonds as some potential sources for pathogenicity.

Introduction

Pompe disease is a rare autosomal-recessive lysosomal storage disorder caused by the deficiency of acid alpha-glucosidase (GAA) which leads to pathological accumulation of glycogen in lysosomes. The severity of Pompe disease and age of onset can be highly variable. The clinical manifestations are dominated by progressive weakness of skeletal muscle throughout the clinical spectrum. The classic infantile form is further characterized by hypertrophic cardiomyopathy. The diagnostic workup for Pompe disease involves a GAA enzymatic activity assay which is necessary because the activity of the GAA enzyme is correlated with the severity and onset of Pompe disease. The infantile, juvenile, and adult onset forms of the disease have an average of 0%, 5%, and 8% residual GAA activity, respectively (Reuser, Hirschhorn, & Kroos, 2014).

The 952 amino acid human GAA enzyme is encoded by the GAA gene. To date, more than 500 different human genetic variants in GAA including missense, nonsense, small or large insertions and deletions, frame-shifting, and splicing changes have been observed and implicated in Pompe disease (Pompe_Center_mutation_database, 2019). While most of these variants are associated with the disease, the functional impact and clinical severity of the variants vary, including several variants that are of uncertain clinical significance. Novel genetic variants will likely be discovered in the GAA gene as more individuals are sequenced. To fully understand the genotype-to-phenotype relationships in Pompe disease, it is therefore critical to characterize the impact of genetic variants in human GAA, particularly on how they affect its enzymatic activity.

With the progress in sequencing of human genomes, several computational methods have emerged within the last decade to predict the impact of genetic variants in human disease. Typically, these variant interpretation methods aim to classify disease-causing pathogenic variants from those that are benign. Some rely predominantly on evolutionary conservation information (Reva, Antipin, & Sander, 2011; Sim et al., 2012), whereas others additionally incorporate physiochemical properties and protein structural data (Adzhubei, Jordan, & Sunyaev, 2013; Carter, Douville, Stenson, Cooper, & Karchin, 2013). Most tools focus on predicting the impact of nonsynonymous changes, but some methods attempt to quantify the mutational effect across the whole genome (Kircher et al., 2014). Recently, meta-predictors have also emerged, that combine scores from several prediction tools using machine learning methods and report an integrated pathogenicity score (Dong et al., 2015; Ioannidis et al., 2016). Instead of directly predicting clinical pathogenicity, yet other tools focus on predicting various intermediate biochemical phenotypes and functional consequences resulting from the genetic variants, including protein stability (Quan, Lv, & Zhang, 2016), RNA splicing (Yeo & Burge, 2004), subcellular protein localization (Laurila & Vihinen, 2011), protein interactions (Betts et al., 2015), and mechanism of action (Li et al., 2009).

Most of the aforementioned methods train broadly in a supervised fashion across large databases of previously characterized human disease-associated and neutral variants. Even though the tools are numerous, the underlying training datasets, features and design principles used are largely homogeneous, and these often suffer from confounding issues of circularity and overfitting arising from overlapping training and evaluation datasets (Grimm et al., 2015).

An implicit assumption made when training classifiers of mutational impact using data from all human genes is that the properties that determine a variant’s deleteriousness are broadly generalizable across all genes. This is certainly reasonable to a degree. For example, amino acids that are highly conserved in evolution are generally intolerant to variants. Similarly, variants that introduce polar sidechains in a protein’s hydrophobic core can destabilize the protein and are generally deleterious. Yet, not all the damaging variants in a particular gene can be comprehensively identified using such general properties alone. Making higher resolution predictions of mutational impact for a particular gene of interest might require incorporating properties that capture the biological context specific to that gene. Some previous efforts have made customized predictions for a gene or gene family of interest (Masica, Sosnay, Cutting, & Karchin, 2012; Torkamani & Schork, 2007) and found that the context of the individual gene, biological pathway and disease can inform the computational models. Still, such gene-specific approaches are less prevalent compared to general pathogenicity prediction tools, in part because of lack of sizable datasets from experimental phenotypic assays targeting specific genes and diseases.

The Critical Assessment of Genome Interpretation (CAGI) experiment aims to evaluate and benchmark computational tools for interpreting the phenotypic impact of genetic variation (Hoskins et al., 2017). One of the challenges in CAGI5 was to make blind predictions of enzymatic activity for >350 different human GAA mutants with experimental data available to the CAGI organizers. In this work, we present a computational framework that integrates common general features with features specific to human GAA gene for predicting the mutational impact on the human GAA enzyme. We applied the framework on the mutations from the CAGI5 GAA challenge to make blind predictions, which were then subsequently assessed. We find that gene-specific features can supplement the predictive capabilities of existing pathogenicity prediction tools. Even though the predicted enzyme activities in GAA were poorer compared to similar enzyme activity challenges in previous CAGIs, we found that incorporating information the full context of the gene can be informative in the interpretation of mutational impact. This was particularly true in mutations that were difficult to predict using existing computational pathogenicity prediction methods alone.

Methods

GAA CAGI challenge test dataset

CAGI5 organizers provided the challenge test dataset, which included cDNA changes of nonsynonymous variants in GAA from an enzyme activity assay performed by BioMarin. Briefly, BioMarin functionally assessed enzymatic activity for 357 nonsynonymous variants in GAA observed in ExAC, most of whose contribution to the disease incidence is unknown. In an immortalized Pompe patient fibroblast cell line with no GAA activity, plasmids containing cDNAs encoding each of the mutant proteins were transfected, and the cells were subsequently lysed, and GAA activity in the lysate was assessed using a fluorogenic substrate. For each variant, the background subtracted mutant enzyme activity was reported as the percent of the background subtracted wildtype GAA enzyme activity. For each mutant, the average normalized % wildtype enzyme activity and standard deviation was calculated from three independent transfections. The GAA CAGI challenge required computational methods to make blind predictions of these % wildtype enzyme activity values, which could then be subsequently independently assessed. Details are provided in the CAGI website (GAA_CAGI5_Challenge, accessed Jan 2019).

Training set

Our approach involved integrating general evolutionary and physiochemical features with GAA-specific features to build a predictive computational model for mutational impact on GAA enzyme activity. For model training, all nonsynonymous variants in GAA gene associated with Pompe disease observed in a locus-specific dababase (LSDB) were collected(Pompe_Center_mutation_database, 2019). Based on the annotated effect in the LSDB, the variants were sorted by severity. The “very severe” annotations were assigned a score of 0 and the “non-pathogenic” were assigned a score of 100. Intermediate values were assigned to the variants with other annotation classes based on the class-wise average enzyme activity observed in for those variants in cited publications from the Pompe center mutation database: very severe: 0, potentially less severe: 3.0, less severe: 5.0, potentially mild: 20.0, presumably non-pathogenic:50.0, non-pathogenic: 100.0 (Kroos et al., 2012; Kroos et al., 2008). For some variants, specific enzyme activity values were available (gathered from Table 2 in (Deming et al., 2017) and the references therein) and these were directly used as activity scores when available. The full collected training dataset and the activity scores are provided in Supp. Table S1. 17 variants in the training set (where enzyme activity values were gathered from previous studies) were also present in the CAGI5 GAA challenge test data set and thus were removed from test set performance assessment.

Table 2:

Performance of the models in classification of putatively pathogenic (<10% wildtype GAA activity) vs. benign GAA mutations.

Binary class threshold = 10% WT enzyme activity (63 positives, 272 negatives)
Model	ROC_AUC	ACC	F1	TPR	FPR	TNR	PPV	NPV	MCC
AA_1	0.61	0.81	0.36	0.29	0.53	0.93	0.47	0.85	0.26
AA_2	0.59	0.8	0.32	0.25	0.56	0.93	0.44	0.84	0.23
AA_3	0.57	0.79	0.27	0.21	0.61	0.93	0.39	0.83	0.17
AA_4	0.63	0.82	0.41	0.33	0.48	0.93	0.52	0.86	0.32
Binary class threshold = 25% WT enzyme activity (125 positives, 210 negatives)
Model	ROC_AUC	ACC	F1	TPR	FPR	TNR	PPV	NPV	MCC
AA_1	0.62	0.69	0.47	0.38	0.36	0.87	0.64	0.70	0.29
AA_2	0.62	0.69	0.47	0.38	0.36	0.87	0.64	0.70	0.29
AA_3	0.6	0.66	0.45	0.38	0.44	0.82	0.56	0.69	0.22
AA_4	0.64	0.67	0.53	0.5	0.43	0.77	0.57	0.72	0.28

Features

Features for predicting the mutational impact of variants could be broadly classified into two categories: general features, which included some of the commonly used evolutionary conservation and functional impact scores; and gene-specific features, which captured features specific to the GAA gene. Most general features were obtained from existing sources whereas most gene-specific features had to be computed anew here.

First, all functional annotations for the GAA gene from dbNSFPv3.3a (Liu, Wu, Li, & Boerwinkle, 2016) were collected and the several evolutionary conservation, functional impact and pathogenicity prediction rankscores were selected as features. For each feature, missing values were replaced by the mean value of that particular feature across all variants for GAA in dbNSFP. Some gene-specific and other general features required protein structural information and were calculated using the mature human GAA determined recently by X-ray crystallography (PDB ID: 5kzw (Berman et al., 2000; Deming et al., 2017)). Beyond the dbNSFP rankscores, the features incorporated into the model included: average distance from two glycogen binding active sites, carbohydrate binding sites, solvent accessibility, residue-wise conservation score, site-wise mainchain-sidechain and sidechain-sidechain hydrogen bonds to critical residues, changes in hydrophobicity/charge/volume upon variant, sequence distance from exon boundaries, protein domain, backbone torsional propensity change, and allele count in the ExAC database (Lek et al., 2016). A full list of the features is provided in Supp. Table S2. Calculations of features specific to this work are described below.

For structural calculations, the Euclidian distances were computed from the Cβ atom of the mutated residue to Cβ atoms of residues participating in biologically relevant sites (glycogen-binding and carbohydrate-binding) in the protein structure. In particular, average distance from the mutant site to the residues forming the first active site 1 (residues 282, 376, 404, 405, 441, 481, 516, 519, 600, 616, 618, 649, 650, 674), second active site 2 (91, 123, 126, 127) and carbohydrate binding site (140, 233, 390, 470, 652, 882, 925) were computed. Sidechain-sidechan and sidechain-mainchain hydrogen bonds were computed using Pymol (DeLano, 2002).

Backbone torsional ϕ,ψ angles of an amino acid are influenced by the identity and geometry of its nearest-neighbor amino acids in the protein chain (Colubri et al., 2006; Jha et al., 2005). Another gene-specific feature that captured perturbation in local backbone geometry, labeled tsp_rmsd, was computed by quantifying the root mean square deviation (RMSD) in the nearest-neighbor dependent backbone torsional angle (Ramachandran) distributions resulting from the variant. From a database of high resolution X-ray crystal PDB structures (resolution < 2.5 Å, homology below 90%), nearest-neighbor dependent Ramachandran (ϕ,ψ) distributions for every possible amino acid trimer had been previously computed and discretized into 72² 5° × 5° bins in a previous work (Adhikari, Freed, & Sosnick, 2012). Individual ϕ,ψ distributions, termed Ramachandran maps, for a site of interest in the protein chain then generated for each amino acid position conditional on both the amino acid and its nearest neighbors.

The effect of the variant of the amino acid backbone torsional angles at the mutant site was quantified, as:

$$tsp\_rms{d}_i={\displaystyle \sum }_{a=i-1}^{a=i+1}RMSD\left(P{\left(r\right)}_a,P{\left(m\right)}_a\right)$$

where P(r)_i and P(m)_i, refer to the nearest-neighbor dependent Ramachandran basin probability vectors (4 basins) at site i for the reference and mutant amino acids, respectively. The 4-basin probability vectors were computed from the Ramachandran maps for the relevant trimer at the mutant site, by discretizing the maps and quantifying the occupancy frequency in the 4 relevant basins: confident α helix, remaining helical region, strand basin and the rest of the Ramachandran map.

From the wildtype human GAA protein structure, we also calculated residue-wise evolutionary conservation scores using the Consurf webserver that leverages phylogenetic relationships between homologous sequences (Ashkenazy et al., 2016; Landau et al., 2005). We used DSSP to compute the solvent accessibility of the reference amino acid in the protein structure (Kabsch & Sander, 1983). Finally, the location of the mutant site in one of the five GAA structural domains (1–136, 137–349, 350–726, 727–820, 821–952), as well as the sequence distance of the mutant to the nearest splice site in the GAA gene were calculated.

Models

Four different computational models were trained. Model AA_3 used predominantly gene-specific features only, models AA_1 and AA_2 used a combination/subset of the gene-specific and general features, and model AA_4 used all the available features. The features selected for each of the four models are provided in Supp. Table S2.

All models treated the prediction task as a regression problem and were thus designed to predict continuous % wildtype GAA enzyme activity values. AA_1, AA_2 and AA_3 used gradient boosted regression tress (GBRT) with 3-fold cross validation for training the model. For each of these models respectively, random 90%, 70% and 90% subsets of the training data were used for training. For each of the three models, GBRT was run 10 times on random subsets of the full training set and the resulting predictions on the CAGI5 test data were averaged to obtain the mean activity scores for that model. Model AA_4 uses all the features and was trained using a single run of XGBoost (Chen & Guestrin, 2016) on the complete training set (Full parameters for the XGBoost function are provided in Supp. Table S3). Additionally, Bayesian hyperparameter tuning was performed in AA_4 using 3 fold cross-validation, with the BayesSearchCV Skopt package.

AA_1, AA_2 and AA_3 were submitted to the CAGI5 GAA challenge. The hyperparameter tuning for AA_4 did not complete before the submission deadline and was not part of the official submission. All models were implemented in Python using the sci-kit learn package (http://scikit-learn.org).

Performance assessment

In total, predictions were made for 335 variants on the CAGI5 GAA test set. Since the challenge involved predicting the impact of the GAA variants on enzyme activity relative to the wild-type human GAA, and our prediction model was designed to report continuous values, we modeled it as a regression problem instead of binary classification. We calculated three different correlation measures for the continuous valued predictions to the experimental % wildtype enzyme activity values: Pearson correlation coefficient (R), Spearman rank correlation coefficient (ρ) and Kendall’s tau-b correlation coefficient (τ). Additionally, we also computed mean absolute error (MAE) of the predictions with the experimental % wildtype enzyme activity values. For the various dbNSFP rankscores, higher values indicate more pathogenic, and thus were assumed to have worse impact on enzyme activity. Therefore, for the correlation coefficients, the absolute value rather than the sign of the correlation will be the relevant metric.

In clinical settings, computational methods are traditionally assessed in their ability to classify benign and pathogenic variants. Since the GAA enzyme activity strongly correlates with affected clinical status in Pompe disease, we also evaluated our prediction in their ability to classify the variants into putatively pathogenic and benign classes. From the continuous valued enzyme activity, binary classification requires choosing thresholds to define the two classes. In severest forms, the residual GAA enzyme activity in patient fibroblasts can be virtually zero, but less severely affected infants can have residual activity in the range of 25–30%. Therefore, we chose two different % wildtype GAA activity thresholds for classification: 10% wild type enzyme activity and 25% wild type enzyme activity. The same corresponding thresholds were used on the predicted values from our models for binary classification assessment.

For the chosen threshold for binary classes, true positives (TP), false negatives (FN), true negatives (TN) and false positives (FP) were calculated. From these, the following metrics of binary assessment were computed as follows:

Accuracy (ACC) = (TP+TN)/(TP+TN+FP+FN)

True positive rate (TPR), or sensitivity = TP/(TP+FN)

True negative rate (TNR), or specificity = TN/(TN+FP)

Positive predictive value (PPV), or precision = TP/(TP+FP)

Negative predictive value(NPV) = TN/(TN+FN)

False positive rate (FPR) = FP/(FP+TN)

Matthew’s correlation coefficient = ((TPxTN)-(FPxFN))/ ((TP+FN)*(TN+FP)*(TP+FP)*(TN+FN))^0.5

Two additional binary classification metrics are reported. The first is the balanced F score, also called the F1 score which computes weighted average of the precision and recall.

F1 = 2 * (precision * recall) / (precision + recall)

The second is area under the receiver operator characteristic curve (ROC_AUC) derived from the predictions for the binary classes.

Results

Overall performance (regression)

We first made direct assessment of the predicted continuous % GAA wildtype enzyme activity values for the GAA variants from the four models against the experimental % wildtype enzyme activity values made available by the challenge data providers. The overall performance metrics for the different models are shown in Table 1. Model AA_4 trained using all the features and XGBoost had the best overall correlation coefficients (R=0.34,ρ=0.34,τ=0.24) among the four models. The model that was composed of primarily gene-specific features (AA_3) had correlation coefficients R, ρ, and τ of 0.25,0.26 and 0.18.

Table 1:

Correlation of the continuous valued enzyme activity predictions with the experimental %wildtype GAA activity quantified using different metrics

Model	Pearson Correlation coefficient (R)	Spearman Rank correlation (ρ)	Kendall Correlation Coefficient (τ)	Mean Absolute Error
AA_1	0.32	0.34	0.24	45.67
AA_2	0.31	0.34	0.24	45.68
AA_3	0.25	0.26	0.18	45.89
AA_4	0.34	0.34	0.24	45.99

Direct comparisons to other existing methods may not be appropriate as the tools typically trained to separate clinically pathogenic variants from benign ones are not optimized for enzyme activity predictions. Yet, mutants in lower enzyme activities should be clinically pathogenic. So, we compared our models to two other recent meta-predictors which integrate several methods. Our model, AA_4 had comparable or better correlation coefficients (R=0.34,ρ=0.34,τ=0.24) compared to REVEL (R=−0.31,ρ=−0.35,τ=−0.24) and MetaSVM (R=−0.23,ρ=−0.28,τ=−0.19). We also compared our predictions to MutPred which had been successfully used for prediction of functional impact in previous CAGI challenges (Figure 1). The correlation scores for MutPred (R=−0.22,ρ=−0.26,τ=−0.18) with the experimental data were lower compared to our models and the meta-predictors. Removing GAA variants where MutPred scores were not available from dbNSFP did not significantly improve the correlations with the experimental data. Out of 245 variants where Mutpred scores were available, the R, ρ, τ coefficients improved slightly to −0.26, −0.30, and −0.21 respectively. An interesting observation was that AA_3, the model using primarily gene-specific features, was least correlated with all the existing tools, highlighting the potential novelty of information contained in the gene-specific features (Supp. Figure S1).

Figure 1:

Scatterplot matrix showing correlation between of one of our prediction models (AA_4) to some other existing pathogenicity prediction tools (REVEL, MetaSVM, MutPred) as well as the experimental % wildtype GAA mean activity (MeanActivity) from the CAGI 5 GAA challenge.

Overall, the performance of our predictions as well as other existing tools on GAA was worse compared to another enzymatic assay challenge from the previous CAGI4. For N-acetyl-glucosaminidase (NAGLU) in CAGI4, the best performing had a Pearson correlation coefficient of around 0.6, with several other methods with correlation coefficient above 0.4 (Katsonis & Lichtarge, 2017).

Overall performance (binary classification)

Next, we evaluated the ability of the models to classify putatively pathogenic GAA variants from benign ones. Based on depletion of enzyme activity typically seen in severe and milder GAA patients, we chose on two different thresholds (<10% wildtype and <25% of wildtype) for enzyme activity to define the putative pathogenic mutants. The AUC_ROC ranged from 0.57 to 0.63, and the overall accuracy ranged from 0.79 to 0.82 for the four models in this task, when using the binary classification threshold of < 10% wild type enzyme activity for the positive (putatively pathogenic) class, with model AA_4 performing better than the rest. When choosing a higher threshold of <25% wildtype enzyme activity for putatively pathogenic variants, the accuracy decreased to a range of 0.66–0.69 in all the models. At the same time, at this threshold, the sensitivity (true positive rate) increased from 0.25–0.33 to 0.38–0.5. Table 2 presents the other metrics of performance of the different models in their ability to classify putatively pathogenic GAA variants.

Characterizing well-predicted and poorly-predicted pathogenic variants

To better understand the correct and incorrect predictions of mutational effect, we investigated the variants in the context of the wildtype protein structure. In particular, we focused on putatively pathogenic variants (where the experimental enzymatic activity was < 10% of the wild type), because majority of the clinically severe cases of Pompe disease have enzyme activity below this threshold. Among these putatively pathogenic variants, those that were correctly predicted (i.e., <10% activity) by the primary model (AA_1) are shown in green and those that were incorrectly predicted (≥10% activity) are shown in red in the human GAA protein structure (Figure 2). Structural data was not available for the first 78 residues and variants in those positions.

Figure 2:

Left: Location in the protein structure (PDB: 5kzw) of the human GAA variants from the CAGI5 GAA challenge with <10% experimental wildtype enzyme activity. Variants correctly predicted to have <10% wildtype activity by our model (AA_1) are shown in green and those failed to be identified are shown in red. Right: For different binary prediction outcome categories for classifying variants that result in <10% experimental wildtype enzyme activity, box plots showing Cβ atom distances of the mutated amino acid site to the protein core

We observed that the correctly predicted putatively pathogenic variants appeared more frequently in the core of the protein than those that were incorrectly predicted. For the putatively pathogenic variants, we computed the distance of the corresponding amino acid Cβ atoms from the protein center of mass. The true positive predictions (correctly predicted to have < 10% wildtype activity) were closer to the core of the protein compared to the false negatives (incorrectly predicted to have ≥10% wildtype activity) (p<1.76e-03, Mann-Whitney-Wilconson (two-sided) test). That the variants disruptive to the core of the protein are deleterious is a general property, which most existing methods that use protein structural features are likely trained to capture. This suggests that pathogenic variants in the surface of the protein, however, may require additional context for accurate predictions.

Gene-specific features aid interpretation of some putatively pathogenic variants

We next explored if the gene-specific features in GAA could elucidate the origin of deleteriousness of some of the poorly predicted mutants. We first investigated what features best correlate with experimental data in mutants with low experimental wildtype enzyme activity (<10%) but where our computational model (AA_1) failed to classify them. For all the numeric features in our work, Figure 3 shows the absolute Pearson correlation coefficients of the different features with the experimental enzyme activity values for this subset of the mutants. Interestingly, in this subset, the features that best correlate with the experimental values are often gene-specific (shown in blue in Figure 3).

Figure 3:

Within variants resulting in <10% wildtype enzyme activity values in the experimental data, absolute Pearson correlation coefficient of the various features with the experimental values. Gene-specific features are shown in blue and general features are shown in green.

Discussion

Results from previous CAGI challenges had indicated that variant impact predictors trained to classify disease-causing variants could potentially be also directly applied for predicting impact of nonsynonymous variants in activity and function of proteins (e.g. NAGLU, CBS, p16), especially when the training of the predictors involved molecular and biochemical features (Pejaver, Mooney, & Radivojac, 2017). In the case of GAA enzymatic activity, it was surprising that most existing tools as well as our predictions performed poorly relative to previous CAGI enzyme functional impact prediction challenges.

Experimental variability

One potential issue could be the variability in experimental measurements of enzymatic activity. For 17 variants in our training set where we gathered enzyme activities from previous studies, Figure 4 shows the corresponding % wildtype enzyme activity values in the experimental data from CAGI5 GAA challenge. While the activity values generally agree, the % wildtype enzyme activities in the CAGI5 GAA challenge were frequently lower for the same variants than values previously reported in literature. Some experimental noise is always expected, but it is unclear what factors accounts for some variants where the disagreements are substantial. For example, previous studies reported % wildtype enzyme activity of the NP_000143.2:p.R11Q variant in GAA to be 115 and therefore non-pathogenic (Kroos et al., 2008). The same variant has an average % wildtype activity of 23.42±4.41 in the CAGI 5 challenge answer key. Alternatively, another variant, D91N, was found to lowers GAA’s affinity for glycogen in previous studies. In the CAG5 experimental data, the variant had near wildtype % enzyme activity (99.55±1.72). These differences could be important and taken into consideration when evaluating and benchmarking computational models of mutational impact.

Figure 4:

For variants where prior enzyme activity data was available (in our training set), the corresponding experimental % wildtype enzyme activity values in the CAGI5 GAAGAA challenge dataset are plotted.

Poorly-predicted vs well-predicted variants

While experimental variability may be a contributing factor to poor performance of the predictions relative to prior enzyme activity prediction challenges, it is more likely that the computational models failed to systematically capture some important biochemical aspects of the GAA enzyme. An interesting observation was that most of the poorly predicted enzyme activity lowering variants often occurred near the surface of the protein. Therefore, one possible explanation could be that some of these surface variants participate in important interactions necessary for enzymatic activity. While general features used in existing tools could well capture information regarding variants that disrupt the core of proteins, perhaps pathogenicity arising from damaging variants on the surface of proteins are not as well-characterized. Important gene-or biological pathway-specific signals may be lost when pathogenicity prediction methods train globally across all genes, which often ignore the specific biological context of individual genes.

Gene-specific models for hypothesis generation regarding origin of pathogenicity

When discussing computational tools, the primary focus typically is on their predictive capabilities, which are obviously important. Nonetheless, computational models also can reveal novel aspects of the disease etiologies and help generate new hypotheses. We present two examples where a deeper exploration of the poorly predicted variants revealed gene-specific properties that could potentially explain the origin of pathogenicity. One of the features that we incorporated in our models was the change in local protein backbone torsional angle propensities resulting from the variants. The geometry adopted by amino acids in a protein chain is highly constrained by the immediately neighboring amino acids. Such neighbor-dependent torsional angle (Ramachandran) distributions have been successfully used previously for prediction of protein folding pathways and tertiary structures (Adhikari et al., 2012; Colubri et al., 2006; DeBartolo et al., 2009). One of the poorly predicted enzyme activity lowering GAA variants was NP_000143.2:p.Pro690Leu (% experimental wildtype activity: 0.53 ± 0.37). Proline residues often disrupt secondary structures like alpha helices and are frequently found at the start of α helices in protein structures (Aurora & Rose, 1998; Richardson, 1981). Pro690 is such an amino acid as observed in the start of an α helix (Supp. Figure S2, left). Supp. Figure S2 (right) shows the change in the nearest-neighbor dependent Ramachandran distributions of the residue Pro690 and the neighboring Glu689 and Ala691 residues, resulting from the NP_000143.2:p.Pro690Leu variant. As expected, the removal of Proline from the helix capping position in lieu of Leucine leads to a dramatic increase in the helical propensity of all the three amino acids. Thus, a hypothesis for the origin of pathogenicity of this variant could be that this rigidity conferring Proline when mutated to Leucine potentially severely disrupts the local geometry of the protein chain. In this example, the specific context in which the Proline appears in the protein structure could be important information. Further confirmatory computational models using molecular dynamics as well as experimental validation would be required to confirm the origin of pathogenicity. Nonetheless, such contextual information is generally lacking in existing pathogenicity prediction tools.

In another example of gene-specific context providing clues for origin of pathogenicity is the NP_000143.2:p.Arg464Ser variant (% experimental wildtype activity: 2.54±1.52). In the wildtype GAA protein, the NH1 atom of the Arg464 sidechain forms a hydrogen bond with the OD2 atom of Asp501 sidechain (Figure 5). The NP_000143.2:p.Arg464Ser variant results in the loss of this hydrogen bond as the Serine sidechain lacks the hydrogen bonding acceptor. Even though this variant is in the surface of the protein, this specific disruption of a sidechain-sidechain hydrogen bond could play a role in the pathogenicity of this particular variant.

Figure 5:

A sidechain-sidechain hydrogen bond is disrupted by a GAAGAA variant. In the wildtype GAAGAA protein, the NH1 atom of the Arg464 sidechain forms a hydrogen bond with the OD2 atom of Asp501. The NP_000143.2:p.Arg464Ser variant results in the loss of the hydrogen bonding partner for Asp502 sidechain.

Both these are examples of severe variants near the surface of the protein that are pathogenic and cause severe form of the disease, but their interpretation likely required specific information not captured by general features used in training most variant interpretation tools.

Limitations of gene-specific models

Incorporating specific biological context in computational variant effect prediction tools is useful. However, one of the main challenges facing gene-specific computational approaches is the lack of large mutational datasets on individual genes. There are well-maintained locus-specific mutational databases for some genes, But most monogenic diseases are rare, and it is unclear when and if most genes will have sizable disease-associated variant databases required for individually customized, but also statistically robust prediction tools. One potential way forward will be to perform high throughput mutational scans in disease-relevant human genes and probe mutational consequences on a wide array of functional assays (Fowler & Fields, 2014; Starita et al., 2015). Recent CAGI 5 challenges (TPMT and PTEN) are starting to address these (Matreyek et al., 2018), but some limitations still exist. As observed in this study, the same functional assays can differ across experiments. Furthermore, designing a functional assay that is most relevant for understanding disease etiology can be challenging. Finally, even when the functional assay is relevant, intermediate biochemical phenotypes may not always correlate fully with clinical phenotype.

Other confounding factors

One important consideration for Pompe disease is that because the disease is recessive, an affected individual’s overall in vivo GAA enzymatic activity will be determined by a combination to two alleles (diplotype) rather than a single allele (haplotype). In a diplotypic context, the effect of a severe allele can be lessened by the presence of a milder allele which may occur in compound heterozygote form in the patient(Reuser et al., 2014). In fact, variants occurring even in the same allele as the pathogenic GAA variant can modulate the severity and onset of Pompe disease through various combinations (Yang et al., 2011). Other genetic and epigenetic background could modulate the activity levels in vivo. Modifying genes, like ACE and ACN3, have been suggested as potentially linked with earlier onset (de Filippi et al., 2010). Multiple forms of human GAA enzyme have also been observed in different populations. The GAA2 form characterized by an NP_000143.2:p.Asp91Asn variant, has lower enzyme affinity for glycogen (Beratis, LaBadie, & Hirschhorn, 1980; Swallow et al., 1989). Another form of the GAA enzyme termed GAA4, characterized by the NP_000143.2:p.Glu689Lys variant is especially prevalent in Asian populations (allele frequency of 0.29). The variable prevalence of such factors across different ancestral groups strongly motivates sequencing studies should sample across a range of different populations when studying the genetics of human diseases.

Lessons from the CAGI5 GAA challenge

It is known that due to interdependent relationships among underlying features and training data, variant effect prediction tools frequently correlate with each other. An important lesson from the CAGI 5 GAA challenge was that the concordance among computational tools does not necessarily translate to more accurate predictions. We observed that the computational tools performed poorly in predicting enzyme activities GAA compared to similar challenges from previous CAGI experiments. Therefore, another important lesson from this challenge is that one can not assume that a well-performing tool for a given gene will apply readily across various other genes. Large benchmarks will be continually needed to validate predictions from tools across a variety of mutational assays and genes. The interdependence among existing tools also prompt for the need of novel approaches to tackle the issue of variant interpretation.

Gene-specific features may provide orthogonal information to those that are already contained in existing prediction tools. Gene-specific explorations can also help identify new features that may be general beyond the particular gene (e.g. helix capping prolines are general features of protein structures, and could be incorporated into existing pathogenicity predictors that already incorporate some protein structural information). The strength of existing global predictors of mutational effects is that they train on a large mutational datasets, and often use powerful machine learning techniques to capture general aspects of mutational pathogenicity mechanisms. Their overall improvement, however, likely comes at the cost of diluting important signals specific to certain genes, gene families and diseases. Alternatively, single genes and gene family specific models can leverage the specific biological context into pathogenicity predictions. However, these methods alone are often data-limited and don’t often leverage the large clinical and genetic datasets across a broad range of diseases. A reasonable way forward may therefore be to explore ways to integrate such gene-specific and general models so that both the accuracy and resolution of the prediction for a disease in question can be improved. Here, we demonstrated one such approach focusing on the GAA enzyme in Pompe disease. Future efforts of pathogenicity prediction in specific diseases may benefit from such hybrid approaches that leverage both general and gene-specific features to predict mutational effects.