INDELpred: Improving the prediction and interpretation of indel pathogenicity within the clinical genome

Summary

Small insertions and deletions (indels) are critical yet challenging genetic variations with significant clinical implications. However, the identification of pathogenic indels from neutral variants in clinical contexts remains an understudied problem. Here, we developed INDELpred, a machine-learning-based predictive model for discerning pathogenic from benign indels. INDELpred was established based on key features, including allele frequency, indel length, function-based features, and gene-based features. A set of comprehensive evaluation analyses demonstrated that INDELpred exhibited superior performance over competing methods in terms of computational efficiency and prediction accuracy. Importantly, INDELpred highlighted the crucial role of function-based features in identifying pathogenic indels, with a clear interpretability of the features in understanding the disease-causing variants. We envisage INDELpred as a desirable tool for the detection of pathogenic indels within large-scale genomic datasets, thereby enhancing the precision of genetic diagnoses in clinical settings.

INDELpred is a machine-learning-based predictive model for discerning pathogenic from benign indels. It outperforms competing methods in computational efficiency and prediction accuracy, emphasizing the importance of function-based features in identifying pathogenic indels. We envisage INDELpred as a desirable tool for the detection of pathogenic indels within large-scale genomic datasets.

Introduction

Advancements in next-generation sequencing technologies have spurred the discovery of numerous genetic variations within the human genome.¹ Among these, insertions and deletions (indels), ranging from a single base to hundreds of bases, are the second most prevalent type of genetic variation, accounting for 15%–21% of all variants.² Although single-nucleotide variants (SNVs) have been the focus of extensive research, indels have not yet received the same level of scrutiny despite their proven link to a wide spectrum of diseases, including epilepsy³ (MIM: 254770), central nervous system anomalies⁴ (MIM: 608281), rare diseases,⁵ and various cancers.⁶ Indels in coding regions can cause frameshift variants or in-frame alterations that may drastically affect protein function and stability, leading to a cascade of pathological effects.⁷ Additionally, the role of indels in non-coding regions is increasingly recognized as the pivotal etiology in gene regulation and mRNA processing,^8,9 which may have far-reaching consequences for cellular function and disease pathogenesis.^10,11,12 However, due to insufficient supporting evidence, these non-coding indels are often overlooked and regarded as variants of uncertain significance (VUSs).¹³ Therefore, interpreting the functional impact of a myriad of indels identified through clinical genomic sequencing is of great importance for personalized medicine yet remains an arduous challenge, underscoring the need for predictive methods to precisely evaluate their pathogenic potential.

Current predictive frameworks for variant pathogenicity predominantly cater to SNVs, with comparatively fewer tools available for assessing indels specifically.^{14,15,16,17,18,19} Existing methods for indel pathogenicity mainly concentrate on the protein-coding genomic regions and are categorically divided into direct and indirect prediction methods. Direct prediction methods are primarily centered on analyzing the structural and functional impacts of indel variants on proteins.^{20,21,22,23,24,25,26} Computational tools such as SIFT¹⁵ and PolyPhen^26,27 harness sequence homology and physicochemical property analyses to assess the potential deleterious consequences of such mutations. Despite their efficacy, these methods are susceptible to yielding false positive results, especially in cases where frameshift variants do not exert a pronounced effect on protein function.²⁸ Conversely, indirect prediction methods augment the assessment of genetic variants by incorporating diverse features, including demographic characteristics,²⁹ meta-predictive factors,³⁰ and additional pathogenicity-associated statistics. Methods like combined annotation dependent depletion (CADD)¹⁸ and consequence-agnostic pathogenicity interpretation of clinical exome variations (CAPICE)¹⁷ employ an expansive array of over 60 genome annotation features to estimate the pathogenic potential of both SNVs and indels. Moreover, meta-predictors such as MetaRNN¹⁶ leverage existing predictors as meta-predictive factors, enhancing the prediction accuracy by integrating outputs from multiple prediction algorithms, although this may escalate the computational time and complexity of the method.

Though some tools, such as CADD, exhibit notable prediction capabilities, their utilities in practical contexts may be compromised by excessive processing durations. Moreover, a recent review study observed that although the meta-predictors have demonstrated effectiveness, the potential circular reasoning within these models may undermine their evaluative reliability.³⁰ Furthermore, a high degree of correlation among features¹⁶ augments redundancy and consequently reduces the interpretability of the model. Additionally, there exists a scarcity of validation efforts using clinical whole-genome sequencing (WGS) data, suggesting that the efficacy of these tools in clinical settings may not be as formidable as indicated by evaluations performed on curated datasets. Beyond the mere quantification of predictive accuracy, the elucidation of the biological relevance of these predictions is of paramount importance. Despite tools like CAPICE demonstrating commendable predictive accuracy in silico, there remains a significant disparity in understanding the biological implications of their perdition outputs.

To address these challenges, we have developed INDELpred, a predictive model characterized by its elevated accuracy, rapidity, and interpretability in the pathogenicity assessment of indels. INDELpred is remarkable for its superior performance, as evidenced by its proficiency across seven distinct evaluation metrics. Notably, INDELpred outpaces existing software in terms of computational speed, delivering swift predictions while necessitating minimal data storage, thereby enabling effective large-scale genomic investigation. Furthermore, upon application on clinical WGS datasets, INDELpred exhibits remarkable robustness by prioritizing indels according to their pathogenicity scores and maintaining a markedly low rate of false positives for the variants deemed pathogenic. Additionally, through anchoring feature importance, INDELpred identifies function-based features as the foremost contributors to its predictive prowess, suggesting the substantial impact of functional regions on the determination of pathogenicity. Overall, we anticipate that INDELpred will yield clinical insights into genomic variants of indels and offer reliable diagnostic utilities for evaluating their pathogenicity, thus having potential implications for helping develop new therapeutic strategies toward personalized precision medicine.

Material and methods

Data collection

The entire data preprocessing process is illustrated in Figure S1. We downloaded indels from the ClinVar,³¹ VKGL,³² HGMD,³³ and gnomAD²⁹ databases, all aligned to the hg19 reference genome. Indels from the ClinVar database dated October 9, 2021, were compiled for INDELpred training (termed the ClinVarTrain dataset), while those as of August 6, 2023, not included in the ClinVarTrain, were used for internal testing (termed the ClinVarTest). To further assess our model, we created two datasets for external testing. The first dataset, collected from the VKGL as of July 2023, was denoted the VKGLTest. The VKGL dataset is a curated database grounded in real-world clinical inquiries and professional oversight. The second, sourced from the HGMD-2017 and gnomAD (v.2.1.1) databases, was referred to as the HGMDTest. Additionally, clinical WGS data aligned to hg38 reference genome were collected from our previous study.³ This dataset encompassed 26 clinically significant indels identified among 30 pediatric patients with epilepsy.

The ClinVar database and clinical WGS datasets were provided in VCF format, whereas the VKGL, HGMD, and gnomAD databases were in tabular format. For the tabular data, we additionally extracted chromosome locations and variant information (reference allele [REF] and alternate allele [ALT]) for indels and converted them into VCF format. We then split multi-allelic sites into separate entries, if existing, and extracted and normalized all indels using bcftools (v.1.9).³⁴ All VCF files use a 1-based coordinate system. For the clinical WGS data, we merged all VCF files from the 30 patients into a single file. To prevent data leakage during model training and evaluation, we excluded sites present in the training dataset from all the testing datasets except the clinical WGS data.

Data annotation

We used the VCF files as input to ANNOVAR,³⁵ leveraging the hg19 version of the refGene (from UCSC RefSeq,³⁶ v.2020-08-17) and gnomAD databases (v.2.1.1) to annotate indels from ClinVar, VKGL, and HGMD databases. For annotating the clinical WGS data, we used the hg38 version. The annotation information provided by the refGene included "Func.refGene," "ExonicFunc.refGene," and "Gene.refGene," while gnomAD offered "controls_AF_popmax." These annotation results were used to derive features for the selected indels and then develop the INDELpred model.

Data preprocessing

The inclusion criteria for an indel were (1) a length between 1 and 100 base pairs (bp) and (2) an annotation result other than “nonframeshift substitution,” “frameshift substitution,” “nonsynonymous SNV,” or “synonymous SNV.” These criteria were applied consistently across all datasets. Additionally, indels initially categorized as benign or likely benign and pathogenic or likely pathogenic were relabeled as benign and pathogenic for this study.

Furthermore, two extra inclusion criteria were further applied to the ClinVarTrain dataset: (3) the site being located on chromosomes 1–22, X, or Y and (4) the site having the CLNREVSTAT status of “criteria_provided, _single_submitter,” “criteria_provided, _multiple_submitters,_no_conflicts,” or “reviewed_by_expert_panel.” The latter one ensured that the indel variants used for model training were of high confidence. For clinical WGS data, the selected variant sites also must be located on chromosomes 1–22, X, or Y.

The HGMDTest dataset was built by integrating two sources. Pathogenic variants were taken from disease-causing mutation sites in the HGMD database, while benign variants were derived from the gnomAD and VariSNP³⁷ databases. VariSNP is dedicated to benign variants, while gnomAD provides population allele frequency (AF) information. We randomly selected a roughly equal number of benign variants from gnomAD with AFs <0.01. To assess the impact of type II circularity,³⁸ which may arise when variants in a gene region are exclusively labeled as either pathogenic or benign, we created an additional validation dataset called HGMDSharedGene. This dataset included only gene regions harboring both benign and pathogenic variants.

After preprocessing all datasets, we cross-checked for conflicting pathogenicity annotations. Due to the significantly small number of conflicting annotation (n = 24), we opted to exclude these entries from the data. Consequently, we finalized the count of benign and pathogenic indels across different datasets for developing and validating the INDELpred model (Tables 1 and S1).

Table 1.

Number of benign and pathogenic indels in different datasets

Dataset category	Dataset name	Benign	Pathogenic	Total
Training dataset	ClinVarTrain	28,104	41,836	69,940
Internal testing dataset	ClinVarTest	16,091	33,294	49,385
External testing dataset	VGKLTest	3,989	3,996	7,985
External testing dataset	HGMDTest	23,667	23,062	46,729
Check type II loops	HGMDSharedGene	15,855	19,336	35,191
Clinical testing dataset	clinical WGS	3,496,179	26	3,496,205

Feature design

To establish INDELpred, we crafted four categories of features, leading to a set of 16 features used for the model development (Table S2).

AF

The variant AF was sourced from the gnomAD database (v.2.1.1), where the feature labeled “controls_AF_popmax” referred to the maximum AF observed within the outbred control population from the gnomAD genomes.

Indel length

The length of the indel was calculated as follows:

$$L_{\text{Indel}}=|N_{\text{ALT}}-N_{\text{REF}}|\text{,}$$

where $N_{\text{ALT}}$ and $N_{\text{REF}}$ denote the number of bp of ALT and REF recorded in the VCF file, respectively. The indel length measured the disparity in the bp counts between the ALT and REF.

Gene-based scores

Gene-based features correlated genes with a series of genetic alterations. The impacted genes encompassed genomic functional regions of functional significance, such as exonic, upstream, intergenic, intronic, non-coding RNA associated, splicing related, and UTRs. Additionally, we examined the typology of mutational events, including frameshift deletions, frameshift insertions, non-frameshift deletions, non-frameshift insertions, start loss, stop gain, and stop loss. To quantify the relevance of these features (i.e., functional regions and mutational events) to pathogenicity, we utilized our training dataset to determine the proportion of pathogenic indels within each functional region of each gene, as defined by the “Gene.refGene" annotation:

$$P_{i\in \left(F_j\cap G_k\right)}=\frac{\sum _{m\in \left(F_j\cap G_k\right)}\mathbb{I}\left({\text{indel}}_m\text{is}\text{pathogenic}\right)}{N_{F_j\cap G_k}}\text{,}$$

where $P_{i\in \left(F_j\cap G_k\right)}$ denoted the gene-based score for indel $i$ situated within the region of feature $F_j$ in gene $G_k$, $j=1,2,..,J$, and $k=1,2,..,K$. $J$ and $K$ are the numbers of unique features and genes annotated for all indels in the training dataset, respectively. $\sum _{m\in \left(F_j\cap G_k\right)}\mathbb{I}\left({\text{indel}}_m\text{is}\text{pathogenic}\right)$ and $N_{F_j\cap G_k}$ represent the number of pathogenic indels and all indels within the region of feature $F_j$ in gene $G_k$, respectively. $\mathbb{I}(\bullet )$ is an indicator function, and $\mathbb{I}\left(x\right)=\{\begin{array}{c}1,\text{if}x\text{is}\text{true}\\ 0,\text{otherwise}\end{array}$.

Function-based scores

Genomic features indicative of tolerance and intolerance to genetic variations provided insights into the potential functional impact of variants on gene expression and protein function. We therefore defined two functional-based features for tolerance to the variations (denoted as “general tolerance” and “exon tolerance”) and another two for intolerance (“general intolerance” and “exon intolerance”), based on the “Func.refGene” (e.g., exonic, upstream, etc.) and "ExonicFunc.refGene" (e.g., frameshift insertion, stop, loss, etc.) annotations.

To quantify the tolerance or intolerance of each function,³⁹ we initially fitted ordinary least squares regression models (statsmodels library, v. 0.14.0) using the training dataset to predict the number of benign and pathogenic indels, respectively, with respect to the number of indel VUSs (neither pathogenicity nor benign) based on the function types as annotated in “Func.refGene” or "ExonicFunc.refGene," considering the use of VUSs could reduce the impact of background noise. We then performed an outlier test on each fitted models, obtaining the studentized residuals to represent the functional-based tolerance and intolerance scores (Tables S3 and S4). Finally, the four features mentioned above with specific scores for a particular indel variant were determined according to the indel’s annotated functional type, thereby called functional-based scores, which offered a measure of the propensity for pathogenicity of the variants.

Development of INDELpred model

Features with missing values were simply imputed with zeros. To establish the INDELpred model, a gradient boosting classifier⁴⁰ (GBDT) was developed. We applied the grid search approach with the stratified 5-fold cross-validation to optimize the hyperparameters of the INDELpred model, which were determined based on the maximum value of the weighted F1 score. INDELpred was implemented upon scikit-learn (v.1.3.0) and Python (v.3.10.12). Consequently, the optimal values of hyperparameters for the GBDT-based INDELpred model were a learning_rate of 0.15 and a min_samples_leaf of 0.01. All remaining parameters were retained at their default settings. After that, we retrained the model configured with the optimal hyperparameters using the whole training dataset to obtain the final INDELpred model.

Feature ranking

We adopted the SHAP⁴¹ (Shapley additive explanations) value (v.0.42.1) to rank the importance of individual features toward predicting the pathogenicity of indels. A SHAP value quantified the marginal contribution of a feature that influenced the prediction probability of pathogenicity. Positive SHAP values were indicative of the increasing risk of pathogenicity, while negative ones were indicative of the decreasing risk. We bootstrap sampled 20% of the indels from the ClinVarTest dataset and calculated the mean absolute SHAP value per feature across these indels to obtain feature importance. This process was repeated 100 times to ensure a robust estimation of feature contributions.

Competing methods

We compared our INDELpred against four predictive models, including CADD,⁴² CAPICE,¹⁷ and VSET.¹⁹ CADD integrated multiple annotations to compute a single score to predict the deleteriousness of both SNVs and indels. CAPICE employed machine learning to predict the pathogenicity of variants unearthed through clinical exome sequencing. Variant effect scoring tool (VEST) leveraged a random forest to score the functional impact of amino acid substitutions and indels. These methods took VCF files as input. The first three methods were downloaded from their respective GitHub repositories, and we adhered to their recommended parameter settings for our comparative experiments. For VEST, we applied its online web service, uploading the VCF file for analysis and deliberately choosing the “VEST-4” option. We then retrieved the resultant scores once the prediction process was completed.

Prediction evaluation on genomic factors

To analyze the impact of various genomic factors on their predictive accuracy and robustness, we partitioned the ClinVarTest dataset into different subgroups following four parameters.

• •Indel length: segmented the dataset into four groups based on the length of indels: "0–10bp," "10bp ∼ 20bp," "20bp ∼ 50bp," and ">50bp".
• •AF: stratified the dataset into four groups based on the AF: "0" (where the variant AF is zero in the gnomAD database or the variant is unrecorded in the database), "0%–0.1%," "0.1%–1%,” and “>1%.”
• •Gain-of-function versus loss-of-function: to explore the differential performance of the model in predicting the functional consequences of indels in gene products between gain of function and loss of function, we classified genes into ONCs and TSGs according to the COSMIC’s Cancer Gene Census.⁴³
• •SharedGenes group: to evaluate the risk of inflated model performance due to type II circularity, we identified a subset of genes within the ClinVarTest dataset that harbored both benign and pathogenic indels. This subset, termed the SharedGenes group, was used to guarantee that the predictive capabilities were not solely a reflection of the models' ability to recognize gene-specific variant patterns.

To investigate the correlation between the data confidence and the predictive accuracy of the models, we segmented ClinVarTest and VKGLTest datasets according to the distinct confidence intervals. For ClinVarTest, we applied the CLINSTAT criteria to assign confidence ratings spanning from 0 to 4 stars based on the number of gold stars associated with the review status. For VKGLTest, data stratification was performed based on the “support” column, which indicates the number of laboratories corroborating the pathogenicity of a specific indel. A higher number indicated a more reliable interpretation of the variant.

Evolution metrics

The performance of the final model was evaluated using the following metrics.

• 1.The area under the receiver operating characteristic curve (AUROC).
• 2.The area under the precision-recall curve (AUPR).
• 3.Matthews correlation coefficient (MCC):

$$\mathit{MCC}=\frac{\mathit{TP}\ast \mathit{TN}-\mathit{FP}\ast \mathit{FN}}{\sqrt{\left(\mathit{TP}+\mathit{FP}\right)\left(\mathit{TP}+\mathit{FN}\right)\left(\mathit{TN}+\mathit{FP}\right)\left(\mathit{TN}+\mathit{FN}\right)}}\text{,}$$

where TP represents true positive, FP represents false positive, TN represents true negative, and FN represents false negative.

• 4.Cohen’s kappa:

$$\mathit{\kappa }=\frac{2\ast \left(\mathit{TP}\ast \mathit{TN}-\mathit{FN}\ast \mathit{FP}\right)}{\left(\mathit{TP}+\mathit{FP}\right)\ast \left(\mathit{FP}+\mathit{TN}\right)\ast \left(\mathit{TP}+\mathit{FN}\right)\ast \left(\mathit{FN}+\right)}\text{.}$$

• 5.Balanced accuracy (BA):

$$\mathit{BA}=\frac{1}{2}\left(\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}+\frac{\mathit{TN}}{\mathit{TN}+}\right)\text{.}$$

• 6.F1 score:

$$\mathit{F}1=\frac{\mathit{TP}}{\mathit{TP}+\frac{1}{2}\left(\mathit{FP}+\mathit{FN}\right)}\text{.}$$

Experimental setting

All experiments were conducted on a workstation with 12 Intel Xeon Gold 5118 CPUs and 188 GB random access memory. We performed experiments within the CPU-based environment and deliberately ran each method on a single thread with a virtual memory allocation of 1 GB for performance comparison.

Results

Overview of INDELpred

We built INDELpred to predict the pathogenicity of indels by leveraging a set of features. Specifically, functional-based features (n = 4) and gene-based features (n = 10) were first derived for indel variants (Figure 1A). By integrating the two individual features of indel AF and indel length, we created a compact yet comprehensive list of 16 features to establish our INDELpred model (Table S2). The GBDT-based INDELpred model was rigorously trained with the ClinVarTrain dataset using the stratified 5-fold cross-validation in conjunction with grid search to ascertain the optimal hyperparameter settings (Figure 1B). Using the optimal hyperparameters, we retained the INDELpred model on the entire ClinVarTrain dataset to finalize the model. The performance of the final INDELpred model was then assessed using a suite of evaluation metrics. To thoroughly evaluate the robustness and reliability in predicting the pathogenicity of indels across different genomic contexts, we tested the model on a variety of datasets, including internal and external testing datasets, as well as a clinical WGS data (Figure 1C).

Figure 1.

Overview of INDELpred development and evaluation

(A) After annotation by ANNOVAR, the function-based features for indel variants were represented by the studentized residuals that were derived from the fitted ordinary least squares regression models, while the gene-based features were calculated by the ratios of pathogenic to all indels for each function in the specific gene regions.

(B) With the four distinct feature categories, the GBDT-based INDELpred model was trained upon the ClinVarTrain dataset using the stratified 5-fold cross-validation. The final INDELpred model was subsequently applied to four testing datasets for the evaluation of prediction accuracy, runtime, and data storage requirements using a set of metrics.

(C) INDELpred model for indel pathogenicity prediction was further validated across various genomic contexts, including stability analysis with different genomic factors, reliability analysis based on the varying support laboratories (confidence levels) of indels, shared gene analysis between HGMD and gnomAD, and the adoption of a clinical WGS dataset with 30 pediatric individuals for practical applicability assessment.

INDELpred enhances the identification of pathogenic indels

We exploited multiple datasets to evaluate the performance of INDELpred in identifying pathogenic indels. These datasets included an internal testing dataset, ClinVarTest, and two external testing datasets, VKGLTest and HGMDTest. For benchmarking purposes, we chose the well-established methods CADD, CAPICE, and the coding region-specific variant scoring tool VEST for comparison. CADD employed a threshold score of 20 to delineate benign from pathogenic variants, whereas the other methods conventionally utilized a threshold of 0.5 for this distinction.

Results revealed that INDELpred achieved comparable or superior prediction accuracy in the comparison with CADD and CAPICE (Figure 2A). Conversely, VEST demonstrated the potential limitations evidenced by the substantial number of missing predictions, with VEST failing to provide predictions for 16,076, 3,003, and 24,550 variants in the ClinVarTest, VKGLTest, and HGMDTest datasets, respectively (Table S5). Furthermore, we randomly selected 80% samples from each testing dataset and assessed prediction performance in terms of the six evaluation metrics. This process was repeated 20 times to investigate the prediction robustness of each method. Results demonstrated that INDELpred was comparable or better than competing methods, as evidenced by its achievement of high values of F1 score (0.979 ± 0.000 for ClinVarTest, 0.928 ± 0.001 for VKGLTest, and 0.985 ± 0.000 for HGMDTest; median ± standard deviation); Cohen’s kappa (0.936 ± 0.001 for ClinVarTest, 0.848 ± 0.002 for VKGLTest, and 0.972 ± 0.001 for HGMDTest); MCC (0.936 ± 0.001 for ClinVarTest, 0.852 ± 0.002 for VKGLTest, and 0.972 ± 0.001 for HGMDTest); BA (0.968 ± 0.000 for ClinVarTest, 0.924 ± 0.001 for VKGLTest, and 0.986 ± 0.000 for HGMDTest); AUROC (0.993 ± 0.000 for ClinVarTest, 0.974 ± 0.001 for VKGLTest, and 0.998 ± 0.000 for HGMDTest); and AUPR (0.996 ± 0.000 for ClinVarTest, 0.966 ± 0.002 for VKGLTest, and 0.998 ± 0.000 for HGMDTest) (Figures 2B, 2C, and S2; Table S6). Despite the presence of genes in the testing dataset that were not used in training, our model still maintained high accuracy (Table S7). Additionally, we qualitatively visualized the distribution of prediction scores that represented the likelihood of pathogenic and benign variants. Results displayed that INDELpred scores were consistently concentrated near 1 for pathogenic variants and 0 in the benign ones across the three datasets (Figures 2D and S3). This comparative analysis underscored the efficacy of INDELpred in distinguishing between benign and pathogenic indels compared to other competing methods.

Figure 2.

Indel pathogenicity prediction performance comparison between INDELpred and the competing methods on the datasets of ClinVarTest, VKGLTest, and HGMDTest

(A) The number of predicted loci by different models for the three testing datasets. The gray color indicates the loci not predicted by the model.

(B) Radar chart of six evolution metrics. Values closer to the periphery represent a score approaching 1, indicating the high prediction performance.

(C) ROC curves with AUROC values yielded by different methods.

(D) Distribution of prediction scores by different models. The horizontal axis ranges from 0 to 1. CADD scores were normalized as ${\text{score}}_{\text{CADD}}\times 0.5/20$, where we chose the value of 20 as the CADD cutoff value.

(E) Time consumption of the three models across the three datasets. Each method was independently executed five times on each dataset to ensure the consistency of the results. The data storage requirement for each method is shown on the x axis. Statistical test: two-sided Mann-Whitney U test with the Benjamini–Hochberg correction. ∗∗p < 0.01 and ∗∗∗∗p < 0.0001. The center line indicating the median and whiskers representing 1.5 × IQR.

We further assessed the computational efficiency and algorithmic complexity for predicting pathogenic indels. VEST was excluded from this evaluation because it is a tool specifically designed for coding region variants, which limits its detection range compared to the other three tools. Consequently, INDELpred exhibited significantly high computational efficiency coupled with minimal data storage requirements (11.7 ± 19.16 min, 30 GB storage), approximately 6- and 28-fold faster than CAPICE (71.18 ± 76.38 min, 84 GB; p < 0.01) and CADD (322.37 ± 197.17 min, 129 GB; p < 0.0001), respectively (Figure 2E; Table S8). Collectively, although CAPICE also showed promising prediction accuracy in terms of some evaluation metrics for the VKGLTest dataset (Table S6), it is the combined consideration of robustness, superior performance, exceptional computational efficiency, and low storage requirements that establish INDELpred as a preferred tool for the accurate prediction of pathogenic indels within genomic data.

INDELpred shows stability for the impact of different genomic factors

Next, to ascertain the predictive capability of INDELpred with respect to various genomic factors, we partitioned the ClinVarTest dataset based on four parameters: indel length, AF, the functional nature of variants (gain of function or loss of function), and the classification within the SharedGenes group. Typically, gain-of-function and loss-of-function variants were linked to the roles of oncogenes (ONCs) and tumor suppressor genes (TSGs), respectively. The SharedGenes group contained the genes that harbored both pathogenic and benign variants. Considering that indels with lengths that are divisible by three represent non-frameshift variants, we also categorized indel length into two subsets based on whether or not it is divisible by three. Results showed that INDELpred maintained a consistently high AUROC in both subsets (Figure 3A). Moreover, AUROC values for all methods tended to decrease with the increase in indel length. However, the decrease in AUROC values for INDELpred was more gradual compared to the other methods (Figure 3B). A similar trend was observed for AF, where AUROC values across all methods declined as AF rose. Nevertheless, INDELpred still maintained the highest prediction accuracy (AUROC = 0.90) even for indels with high AF (>0.01) (Figure 3C). These findings suggested that INDELpred was more robust and stable in handling long and common indels, which can be challenging for other prediction algorithms. Besides, the analysis also revealed that all methods exhibited better performance in TSGs than ONCs (Figure 3D), indicating that these methods were more adept at identifying loss-of-function variants. Additionally, the performance of INDELpred, along with those of CAPICE and CADD, was not affected by the presence of type II loops, as the AUROC values remained consistent (Figure 3E). This was further supported by consistent results across various evaluation metrics in the HGMDSharedGene dataset, which contains the same genes as both pathogenic and benign variants and maintained comparable metric accuracy to that observed in the HGMDTest dataset (Figure 3F). VEST exhibited the lowest prediction performance across all four genomic factor parameters.

Figure 3.

Evaluation of INDELpred for the impact of different genomic factors

(A–E) AUROC results assessed for each partition subset of the ClinVarTest dataset based on genomic factors of whether or not the indel length was divisible by three (A), indel length (B), AF (C), groups of TSGs and ONCs (D), and groups of shared genes (E). The number of indels in each subset is indicated on the top accordingly.

(F) Radar chart of the six metrics valued using the HGMDSharedGene dataset.

(G) AUROC results with respect to different levels of CLINSTAT credibility in ClinVarTest (left) as well as different support laboratories in VKGLTest (right).

Furthermore, we partitioned the ClinVarTest and VKGLTest datasets to examine the prediction performance with respect to the varying levels of confidence. As expected, for the ClinVarTest dataset, a general upward trend in AUROC values was observed across all methods as data credibility increased. Remarkedly, INDELpred outperformed the other methods at all levels of confidence, achieving impressively high yet stable AUROC values of 0.993, 0.995, and 0.996 for confidence stars ≥0, ≥1, and ≥2, respectively (Figure 3G, left; Tables S9 and S10). In the VKGLTest dataset, INDELpred and CAPICE demonstrated impressive stability. INDELpred, in particular, maintained comparable high AUROC values of 0.975, 0.994, and 1.000 for support levels ≥1, ≥2, and ≥3, respectively (Figure 3G, right; Tables S11 and S12). These results suggested the high accuracy and stability of INDELpred in predicting pathogenic indels under different genomic factor and confidence levels.

INDELpred improves efficiency in identifying pathogenic indels within clinical WGS data

Next, we applied INDELpred and the competing methods to the clinical WGS data collected from a cohort of 30 individuals, which harbored approximately 3 million unique indel variants. Among these, 26 distinct variants were strongly corroborated as etiological indel variants (Table 1). INDELpred and CAPICE successfully processed all variants in the analysis, while CADD failed to complete three individuals due to exceeding the predefined computational resource constraints. Notably, INDELpred displayed exceptional efficiency with a median processing time of 52 ± 45 min, significantly faster than CAPICE (228 ± 204 min, p < 0.0001) and CADD (1,888 ± 1,314 min, p < 0.0001) (Figure 4A; Table S13). Owning to the profoundly imbalanced dataset where the benign variants far exceed the pathogenic ones, we calculated the AUPR to measure the prediction performance in conjunction with the AUROC. Compared to the competing methods, INDELpred reached the highest AUPR of 0.506 and AUROC of 0.990 (Figures 4B and S4).

Figure 4.

Comparison of indel pathogenicity prediction performance between INDELpred and the competing methods on the clinical WGS dataset

(A) Computational time taken by the three methods for the clinical WGS dataset collected from 30 pediatric individuals. Statistical test: two-sided Mann-Whitney U test with the Benjamini-Hochberg correction. ∗∗∗∗p < 0.0001. Each box in the boxplots corresponds to the interval between the 25th and 75th percentile (interquartile range, IQR) with the center line indicating the median and whiskers representing 1.5 × IQR.

(B) Precision-recall curves with AUPR values for each predictive model.

(C) Curve plot showing sensitivity of pathogenic indel in the top-k variants ranked by different prediction methods for varying k values.

(D) Bar plot showing sensitivity of individual in top 30, top 100, and top 150 variants ranked by different prediction methods.

(E) Prediction scores yielded by different predictive models for the 26 disease-causative indels. CADD scores were normalized as ${\text{score}}_{\text{CADD}}\times 0.5/20$, where we chose the value of 20 as the CADD cutoff value. Each box in the boxplots corresponds to the interval between the 25th and 75th percentile (interquartile range, IQR) with the center line indicating the median and whiskers representing 1.5 × IQR.

To assess the predictive power of INDELpred in pinpointing pathogenic indels, we prioritized and ranked the variants based on their predicted pathogenicity scores. We calculated the sensitivity of pathogenic indels that measured the proportion of true positive pathogenic indels in the top-k variants as ranked by the prediction method. Compared to the other methods, INDELpred demonstrated markedly higher achievements in the sensitivity of pathogenic indels for the varying k values (Figure 4C). For example, it reached 92.3% (24/26) within the smallest number of top variants (k = 136), surpassing the performance of the other methods. In addition, we further calculated the sensitivity of individuals with true positive pathogenic indels in the ranked top-k variants. Results showed that INDELpred outperformed the other methods (Figure 4D). It enabled us to identify pathogenic indels in 56.7% (17/30), 90% (27/30), and 93.3%(28/30) of individuals within the top 30, top 100, and top 150 variants, respectively. Additionally, the distribution of the predicted pathogenicity scores for the 26 disease-causing indels showed the highest median value (0.997) that was achieved by INDELpred (Figure 4E). Considering the high accuracy and computational efficiency demonstrated by INDELpred in clinical whole genome sequencing, our tool showed superior potential for application in sifting through large-scale genomic data to accurately identify pathogenic InDels within clinical contexts.

INDELpred illustrates the impact features on pathogenicity prediction

We further explored the relevance of different features used in the INDELpred model, which were measured by SHAP values. We first examined the correlation between features, and the results showed that most features exhibited minimal interdependence, with the exception of function-based features (Figure 5A). The relatively high correlation within function-based features was likely due to the presence of benign and deleterious indicators. With SHAP values, we identified the top five most important features, including exon intolerance, AF, general tolerance, general intolerance, and exonic (Figure 5B). It is worth noting that the exonic feature exhibited the most importance among other gene-based features, suggesting its critical role in predicting pathogenic indels. To further quantify the contribution of each feature type toward the INDELpred’s ability to predict pathogenic indels, we compared the relative importance of the four feature types. Result showed that function-based features and AF exhibited the two largest contribution to the prediction of pathogenicity (Figure 5C).

Figure 5.

Analysis of pathogenicity-related factors through the lens of features utilized by INDELpred

(A) Spearman correlation coefficient matrix of the 16 features used by INDELpred.

(B) The contribution of each individual feature, represented by feature importance, toward the prediction of indel pathogenicity by INDELpred model.

(C) Relative contribution of each of four feature categories toward the prediction of indel pathogenicity by INDELpred model. Each box in the boxplots corresponds to the interval between the 25th and 75th percentile (interquartile range, IQR) with the center line indicating the median and whiskers representing 1.5 × IQR.

(D) Comparison of AUROC results across three datasets for AF with threshold-free and threshold-based values.

(E) Hierarchical clustering analysis of 16 individual features. All values of individual features were Z scored.

The AF quantified the prevalence of the indels within a population. The lower AF was generally indicative of a more deleterious variant, which was consequently less likely to persist through generations. The American College of Medical Genetics and Genomics (ACMG) guideline suggested that AF can serve as a threshold to enrich for pathogenic variants.⁴⁴ However, recent studies advocated for the adoption of a precise AF value rather than a broad threshold.⁴⁵ In light of these findings, we evaluated the impact of incorporating AF as a precise value versus using a threshold on INDELpred’s performance. Our analysis revealed an enhanced model performance when AF was treated as a specific value (Figure 5D), confirming its significance in the accurate determination of variant pathogenicity. Furthermore, a hierarchical clustering analysis using all 16 features also demonstrated their collective efficacy in discriminating between benign and pathogenic indels (Figure 5E).

Discussion

We have developed INDELpred, a machine-learning-based model characterized by its computational efficiency and remarkable accuracy that surpasses other established methods in the prediction of pathogenic indels across a variety of evaluation datasets. INDELpred distinguished itself primarily in two aspects. Firstly, it outpaced the competing methods without compromising performance accuracy. Importantly, INDELpred has demonstrated its potential for clinical utility in large-scale genomic studies, which is essential in the context of precision medicine. Secondly, it incorporated a range of compact yet informative features, including AF, indel length, function-based features, and gene-based features that facilitated the elucidation of their relevance in the pathogenic potential of indels. Through comprehensive analysis, INDELpred showed its capacity to rapidly unravel the genetic underpinnings of disease-causing variants, thereby improving the interpretation of genomic data for accurate diagnoses.

Variant interpretation tools rarely prioritize runtime optimization as much as the initial variant calling steps, partly due to SNV prediction tools relying on precomputed databases to expedite the process.^45,46,47 However, when they encounter more complex variants like indels, SVs, and copy-number variations (e.g., annotSV,⁴⁸ X-CNV⁴⁹), the computational burden can be overwhelming, and thus, runtime becomes a more pressing concern. INDELpred addressed this concern by leveraging a streamlined set of 16 features highly pertinent to indels, including function-based features that significantly contribute to the prediction outcomes. The post-annotation was only based on essential databases such as gnomAD for AFs and refGene for gene information. Compared to existing tools that can predict both SNVs and indels, such as CADD and CAPICE, INDELpred managed to save disk space and reduce the computational load significantly, making it highly suitable for genome-wide analyses in clinical settings as well as for processing large public databases. Another challenge that constrains the utility of computing tools in genomics is the accuracy of their prediction.⁵⁰ The highest AUROC value does not necessarily equate to an optimal performance, particularly in the context of imbalanced datasets used for establishing predictive models.⁵¹ Here, we have systematically assessed INDELpred along with other competing methods by employing an array of evaluation metrics such as AUROC, MCC, and Cohen’s kappa. These metrics gauged the discriminative ability and overall accuracy of the model. Additionally, metrics like the F1 score and average precision score were essential for harmoniously balancing precision and recall, particularly critical in the prediction of rare variants.

In data selection, we have considered the potential for type I and type II loops, that is, data leakage or the presence of individual genes being exclusively either pathogenic or benign, which can artificially inflate model performance.^38,52 Since we did not utilize the results from the existing tools, data leakage can be heavily mitigated through the strict separation of training and testing datasets, with the testing dataset remaining entirely undisclosed during the feature preprocessing of the training dataset.^45,53 Other tools such as CAPICE may encounter the challenge of type I loops due to their reliance on previously developed tools.¹⁷ Furthermore, INDELpred alleviated the risk of type II loops using the SharedGene subsets from ClinVarTest,^54,55 which comprised only 1,712 loci. Given the potential bias introduced by the relatively small number of loci (1,712) in the SharedGene subsets from ClinVarTest, we further incorporated the HGMDSharedGene dataset, which includes a total of 35,190 loci, all covered by SharedGene. This inclusion broadened our dataset and diminished the likelihood of falsely elevated accuracy resulting from type II loops.

The features used in INDELpred were primarily calculated based on the pathogenic and benign nature of indels in the training dataset, thus offering clear interpretability. Among them, the function-based features contributed the most toward predicting pathogenic indels, surpassing the influence of gene-based features. This was largely attributed to the exertion of the ClinVar database for training, where the primary determinant of a variant’s classification was its association with disease causation. Genetic variations, encompassing both coding and non-coding regions, can have a profound influence on protein levels and function.^56,57 Variants that altered amino acids may disrupt protein stability or specific functions such as molecular interactions or enzymatic activities. Similarly, variations in regulatory elements can affect gene expression, while changes in splicing motifs can result in substantial modifications to protein sequence and structure.⁵⁸ These alterations can lead to either a loss or gain of protein function, which may ultimately cause disease. In contrast, the impact of gene-specific changes on pathogenicity is often related to particular protein functions or types of disease.⁵⁹ This aspect was more challenging to capture by INDELpred, as the current training dataset does not explicitly account for the diversity and complexity of gene-specific effects. Therefore, a database that incorporates disease types or severity as quantitative measures is needed for a more nuanced representation of the impact of genes on indel pathogenicity.

The prevalence of polymorphisms within a population was a key indicator for evaluating genetic variants. In the INDELpred model, AF emerged as a singular yet highly influential feature, ranking as the second contribution toward the assessment of indel pathogenicity. This underscored its significance in the determination of a variant’s potential impact on health. Nevertheless, the utility of this metric was somewhat diminished when analyzing rare variants, necessitating the integration of additional indicators to achieve a more robust analysis. Moreover, guidelines from ACMG and Association for Molecular Pathology (AMP) posited that a minor AF greater than 5% typically denoted a benign nature of the variant.⁴⁴ Despite this, previous studies have suggested that the exact value of AF presented a more profound influence on model accuracy than a binary categorization based on a threshold.⁴⁵ Our study supported this view, confirming that the specific AF value markedly affects INDELpred’s predictive performance. However, our dataset currently exhibited a paucity of information regarding AF in the East Asian population.²⁹ With the acquisition of more comprehensive data, we anticipated the development of a model that was finely tuned to the genetic characteristics of different regions, thereby enhancing the precision of pathogenicity predictions for diverse populations.

One primary aim of variant deleterious prediction methods was to improve the interpretation of variants called by clinical sequencing.⁴⁴ These prediction methods have been widely applied to the analysis of the WGS or whole-exosome sequencing data in clinical settings.^60,61 However, many methods tended to produce low specificity, resulting in the misclassification of a large number of neutral variants as deleterious. In the clinical context, the imbalance between the number of pathogenic and non-pathogenic variants suggested that setting a stringent threshold for classification is not always practical or informative. Instead, methods that can stratify variants based on their likelihood of being pathogenic can be more useful. For example, the INDELpred method demonstrates a promising approach by identifying pathogenic loci for 17 out of 30 individuals (56.7%) among the top 30 loci. Comparatively, INDELpred shows superior sensitivity and specificity to other methods, suggesting that it is a more effective tool for discerning the most likely pathogenic loci in a clinical setting.

Several limitations should be acknowledged in the present study. First, the lack of databases with quantitative indicators for disease severity and individual disease likelihood hampers the ability to make finely tuned predictions about health outcomes.⁵⁸ Furthermore, INDELpred thereby suffers from interpreting the cumulative effects of small changes across multiple genes, known as polygenic contributions, and how these changes collectively influence disease development. In addition to the need for quantitative data, incorporating findings from genome-wide association studies⁶² and protein structural features⁶³ can provide valuable insights into the genetic basis of diseases and, thus, improve the assessment of variant pathogenicity. Additionally, it should be noted that our current clinical sample size is relatively small (30 patients). A larger clinical cohort could offer more robust and dependable evidences of the prediction capabilities in a clinical setting. Future studies with expanded clinical sample sizes are necessary to further validate and enhance the clinical applicability of INDELpred.

In summary, we provided the INDELpred model, a robust tool for the prediction of indel pathogenicity that exhibits high efficiency and notable performance by capitalizing on interpretable features. Our analysis reveals that functional-based features are the most significant contributors to the accurate identification of pathogenic indels. INDELpred offers critical genomic insights into indels, enhancing our ability to not only elucidate indels implicated in disease causation but also detect deleterious indels with precision and accuracy. Consequently, INDELpred holds promise for refining diagnostic processes and facilitating the development of personalized therapeutic approaches.

Data and code availability

The code for INDELpred model development and validation is available at https://github.com/yilin-wei98/INDELpred. The raw clinical WGS data, which belong to a previous article,³ were deposited in the CNSA with accession code CNP0000788. The corresponding VCF files may be available from the authors with reasonable request.

Web resources

ANNOVAR, https://annovar.openbioinformatics.org/en/latest/

CADD 1.6 post-release 1, https://cadd.gs.washington.edu/

CAPICE v.5.1.1, https://capice.molgeniscloud.org/

ClinVar, https://www.ncbi.nlm.nih.gov/clinvar/

gnomAD, https://gnomad.broadinstitute.org/

HGMD, https://www.hgmd.cf.ac.uk/ac/index.php

INDELpred, https://github.com/yilin-wei98/INDELpred/

OMIM, https://www.omim.org/

VariSNP, http://structure.bmc.lu.se/VariSNP/

VEST-4, https://cravat.us/CRAVAT/

VKGL, https://vkgl.molgeniscloud.org/