Benchmark of computational methods to detect digenism in sequencing data

Abstract

Digenic inheritance is characterized by the combined alteration of two different genes leading to a disease. It could explain the etiology of many currently undiagnosed rare diseases. With the advent of next-generation sequencing technologies, the identification of digenic inheritance patterns has become more technically feasible, yet still poses significant challenges without any gold standard method. Here, we present a comprehensive overview of the existing methods developed to detect digenic inheritance in sequencing data and provide a classification in cohort-based and individual-based methods. The latter category of methods appeared the most applicable to rare diseases, especially the ones not needing patient phenotypic description as input. We discuss the availability of the different methods, their output and scalability to inform potential users. Focusing on methods to detect digenic inheritance in the case of very rare or heterogeneous diseases, we propose a benchmark using different real-life scenarios involving known digenic and putative neutral pairs of genes. Among these different methods, DiGePred stood out as the one giving the least number of false positives, ARBOCK as giving the greatest number of true positives, and DIEP as having the best balance between both. By synthesizing the state-of-the-art techniques and providing insights into their practical utility, this benchmark serves as a valuable resource for researchers and clinicians in selecting suitable methodologies for detecting digenic inheritance in a wide range of disorders using sequencing data.

Introduction

A paradigm in the study of human diseases is that rare diseases follow a monogenic mode of inheritance, according to which a single variant in a heterozygous or homozygous state alters a single gene to cause the disease. The widespread deployment of high-throughput sequencing allowed the diagnosis rate of rare diseases to reach 30 to 50% of cases, most of which corresponding to monogenic cases [1, 2]. Part of this lack of diagnosis can be due to the large focus given to coding regions of the genome, although variants impacting regulatory regions can be causal of rare diseases [3–5]. In the last 30 years, several diagnoses of rare diseases have also been identified following complex genetic inheritance patterns, thus questioning the monogenic and single-variant inheritance paradigm [6]. Here, we will focus on searching for complex genetic inheritance, and especially for the digenic model of inheritance, which we will also refer to as digenism or Digenic Inheritance (DI). For DI, the presence of damaging variants in two different genes is necessary for the disease to develop [7]. Importantly, in this model, each variant alone is not sufficient for the disease to manifest. If variants in more than two genes are necessary to express the disease, it will be referred to as oligogenism or oligogenic inheritance.

DI is a special form of interaction and initially, the term epistasis was used to describe some cases of digenic inheritance [8]. Epistasis refers to the interaction between genes where the presence of a variant at one locus modifies the effect of the variant at another locus [9]. It covers many different situations with various effects at each locus, including situations of monogenic inheritance with modifier gene effects where a variant in one gene is sufficient to cause the disease but severity is modulated by a variant in another gene [10, 11]. Here, we will only focus on situations of true DI where each variant alone has no impact (no marginal effect) and it is the interaction of the two variants that causes the disease.

The first description of true digenic inheritance in the literature was for retinitis pigmentosa in 1994. It involved two genes encoding proteins found in photoreceptors: ROM1 and peripherin/RDS [12]. Data from three pedigrees and the known interaction between the proteins produced by the two genes supported this conclusion. There were only 6 other examples of DI described until 2001, when more reports started to be published each year. Several of these reports were about Bardet-Biedl (BBS) syndrome [13–16], which is one of the diseases currently associated with the most digenic or oligogenic combinations. The authors based their conclusion again on pedigree data [13], and proposed that three variants in two known BBS genes (BBS2 and BBS6) could explain the phenotype in four of the families. Overall, digenism has been involved in the etiology of several rare diseases [17–20]. In addition, digenism is proposed as an hypothesis for some of the currently unsolved rare disease cases [11].

To make information on currently described digenic and oligogenic diseases more accessible and facilitate further investigations in the field, the database DIDA [21] (DIgenic diseases DAtabase) and its successor OLIDA [22] (OLIgenic diseases Database) were created. DIDA and OLIDA are comprehensive curated resources dedicated to documenting digenic and oligogenic diseases reported in the scientific literature to date. It can be noted that OLIDA encompasses all of the gene pairs present in DIDA. These databases play a crucial role in cataloging and organizing information about digenic and oligogenic combinations, like their associated variant coordinates, gene names, disease IDs and literature references. There are currently 219 diseases featured in the OLIDA database and thus associated with a digenic or oligogenic mode of inheritance.

Despite great advances in next-generation sequencing (NGS) technologies, the detection of DI remains a challenging issue. Only a relatively small number of digenic or oligogenic combinations have been described in the literature to this day (1808 combinations involving 1200 different genes according to the OLIDA database as of July 2024). A common feature of the great majority of reports on DI is the reliance on familial data from multiple pedigrees and a focus on DI involving at least one gene already known as implicated in the disease. This type of approach can be limited in the case of very heterogeneous or very rare diseases, for which familial data is scarce, and also prevents the discovery of new genes involved in digenic diseases. Methods for rare disease variant discovery used on individual exomes or genomes are also not suited for digenic pairs discovery. These approaches are designed for highly-penetrant monogenic disorders and include variant filtering and classification through ACMG guidelines [23]. However, variants implicated in digenic diseases may not be as rare or predicted pathogenic taken separately, and may act together through diverse molecular mechanisms, including protein-protein interactions, shared signaling pathways or complementary functional roles within cellular processes to cause disease [24]. Considering these challenges, several computational approaches have been developed over the years to tackle the issue of DI detection without relying as much on familial data and knowledge about the disease of interest [25–34].

Here, we present a comprehensive overview of the modalities, strengths and limitations of these computational methods published to date to detect DI in NGS data. We classified methods depending on the type of data taken as input: a cohort of patients (cohort-based methods) or a single patient (individual-based methods), with or without a phenotypic description of the patient. Individual-based methods not needing a phenotypic description as input appeared as the most applicable in the case of very rare or heterogeneous diseases, but determining the usefulness of the different methods in this category was complex. We thus performed a benchmark of individual-based methods to detect DI which do not need phenotype information as input. We devised an evaluation protocol to test the performance of each method to detect digenism compared to neutral gene pairs (benchmarking scenario 1), as well as their ability to distinguish between digenic and monogenic inheritance models (benchmarking scenario 2). Our evaluation protocol leverages known digenic pairs from OLIDA, neutral variants from the French general population FrEnch EXome (FREX) project, and pathogenic variants from the ClinVar [35] database. This allowed us to point out the methods with the best sensitivity, specificity and overall predictive power. The results from this benchmark will help users to make informed decisions about their choice of methods to detect DI depending on their needs and available data.

Currently available computational methods to detect digenic inheritance in sequencing data

This work is focused on computational methods to detect DI applicable to sequencing data, that do not need familial data to make an inference and do not restrict the analysis to known disease genes or known digenic disease genes. We categorized existing methods in two categories, depending on the type of data they take as input: cohort-based methods and individual-based methods (Table 1). Cohort-based methods leverage different statistical modeling to find burdens of rare variants or enrichments in gene pairs in a group of patients, with or without comparison with control data. Individual-based methods provide a score for variant pairs or gene pairs that summarizes how likely pathogenic the pair is. These methods leverage machine learning (ML) algorithms to make such inference, which we will detail in subsequent sections. Among individual-based methods, two methods (OligoPVP and Hop) take as input the phenotype or symptoms description of the patient in addition to genetic data : in the following, ML-sympt-info refers to the symptom-informed methods, as opposed to ML-sympt-agno referring to symptom-agnostic methods.

Table 1.

Main characteristics of the computational methods developed to detect DI; SS: percentage of the individual predictors agreeing about the pathogenic class; CS: median probability among individual predictors that the variant pair is pathogenic.

Category	Name	Availability	Input type	Output type	Input patient phenotype	Reference
Cohort-based methods	Digenic Method	R scripts	Variant list	p-values for each variant-gene or gene-gene pair	×	Kerner et al. [29]
	RareComb	R package	Variant list	list of statistically significant variant combinations that meet the user-specified input criteria	×	Pounraja et al. [31]
Individual-based methods	VarCoPP	Online platform (ORVAL)	Variant list	final class: “neutral” or “disease-causing” SS: score [0-100] CS: score [0-1]	×	Gazzo et al. (2017) Renaux et al. [27]
	DiGePred	Precomputed list for all gene pairs	Gene pairs	score [0-1] for gene pairs	×	Mukherjee et al. [30]
	DIEP	Precomputed list for all gene pairs	Gene pairs	score [0-1] for gene pairs	×	Yuan et al. [32]
	VarCoPP2.0	Online platform (ORVAL)	Variant list	score [0-1] for variant pairs	×	Versbraegen et al. [33]
	ARBOCK	Command line tool	Gene pairs	score [0-1] for gene pairs	×	Renaux et al. [28]
	OligoPVP	Command line tool	Variant list	score [0-2] for variant pairs	√	Boudellioua et al. [25]
	Hop	Command line tool	Variant list	score [0-1] for variant pairs	√	Gravel et al. [34]

Cohort-based methods to detect digenic inheritance

We classified two methods as cohort-based methods as they allow the exploration of all pairs of genes harboring at least one candidate variant in a cohort of patients. The first method is the Digenic Method [29] (DM) that adapts the idea of a burden test to detect true digenism or to find a modifier variant for known monogenic conditions. Rare candidate variants selected based on different parameters like their predicted impact or Minor Allele Frequency (MAF) are collapsed within the unit of a gene. Individuals are “carriers” for a specific gene if they carry at least one rare candidate variant in the gene. Then, a case-only logistic model is derived where the carrier status at the first gene is modeled as a function of the carrier status at the second gene, potentially including additional covariates. In this model, it is also possible to test for the modifying effect of common variant by considering the carrier status at these variants individually. R and bash scripts are available to run the method, but not in the form of an R package. Input files include two VCF files one with the genotypes of individuals at the selected rare variants and one with the genotypes of individuals at the selected common variants.

The second cohort-based method is RareComb [31]. It combines the Apriori algorithm [36] and statistical modeling to detect combinations of genes associated with a phenotype. The Apriori algorithm enumerates, in cases and controls independently, all the combinations of rare deleterious variants seen in at least one individual. Then, a binomial test is used to compare the observed frequency of each combination against the frequency expected under the hypothesis of independence, in cases and controls independently. In so doing, RareComb identifies combinations that are enriched in cases and not in controls. Unlike the methods described previously, RareComb is available as a R package and takes as input a sparse Boolean matrix instead of a VCF file. This sparse matrix is created by the user and encodes information for individuals in rows and genes in columns: 0 codes for the absence and 1 for the presence of a rare deleterious variant (as defined by the user) in the gene for an individual.

Overall, cohort-based methods to detect DI make a number of assumptions on the type of variants selected for the analysis (prefiltering steps in the DM and RareComb). They also both need genetic data for many patients and/or controls to be applied with enough statistical power, which can be a limitation in the case of very rare or heterogeneous diseases. For instance, in the case of RareComb, the authors illustrate the method on two cohorts, one with 6189 affected individuals and one with 1878 affected individuals. For the Digenic Method, the authors implement the method on 191 craniosynostosis probands.

Individual-based methods to detect digenic inheritance

Methods not requiring patient phenotype as input (ML-sympt-agno)

Individual-based methods were developed to address the challenge of detecting DI in sequencing data from another angle. We will cover in this section the methods that do not take as input the phenotypic or symptom characteristics of the patient. All of these methods involve ML algorithms, which became prominent in recent years [37], so we will refer to them as ML-sympt-agno methods. ML-sympt-agno methods learn from features associated with known digenic pairs described in the literature (cataloged in DIDA or OLIDA) and with neutral pairs observed in individuals from the general population. ML-sympt-agno methods then predict if a new pair of variants or genes is likely to be pathogenic. The characteristics of these ML methods are summarized in Table 2. Most of them use a Random Forest (RF) algorithm, with slight differences in the exact model used (see Table 2).

Table 2.

Parameters of the ML-sympt-agno methods to detect DI; RF, Random Forest.

Method	Model	Positive training set	Negative training set	Features
VarCoPP	RF ensemble predictor	DIDAv1	1000 Genomes Project	11 features
DiGePred	RF classifier	DIDAv1	Unaffected relatives Undiagnosed Diseases Network	12 features
DIEP	RF ensemble predictor	DIDAv1	Random pairs filtered using 1000 Genomes Project	17 features
VarCoPP2.0	Balanced RF	OLIDA	1000 Genomes Project	20 features
ARBOCK	Decision set	OLIDA	1000 Genomes Project	BOCK knowledge graph

A main characteristic of ML-sympt-agno methods is their positive and negative training sets. VarCoPP, DiGePred and DIEP were all trained on positive pairs from DIDAv1. DIDAv1 contains 200 digenic variant pairs, which fall into 140 unique gene pairs. VarCoPP considers pairs of variants and used the 200 DIDAv1 digenic variant pairs as a positive training set. DiGePred and DIEP consider pairs of genes and thus used the 140 DIDAv1 digenic gene pairs as a positive training set. VarCoPP2.0 was the only RF method trained on OLIDA, which included the previously described DIDA variant pairs and some additional ones. In OLIDA, each variant pair is given a score to indicate the strength of evidence of oligogenicity. This score, called FINALmeta score, summarizes the genetic and functional evidence supporting the disease-causing character of the combination in the literature. VarCoPP2.0 was trained on the 301 digenic pairs from OLIDA with evidence of oligogenicity (FINALmeta score ≥ 1) [38]. Regarding the negative training set, VarCoPP and VarCoPP2.0 selected respectively 100,000 and 150,500 random variant pairs from the 1000 genomes project [39]. DIEP also used the 1000 Genomes Project data for negative gene pairs. First, DIEP’s developers selected randomly 50,000 gene pairs from all possible genes (Random set) as well as 13,390 gene pairs obtained by randomly combining DIDA unique genes, excluding true DIDA pairs (DIDA_NI set). Then, these two negative gene sets were filtered by only keeping gene pairs which involved pairs of variants with the following criteria: rare (frequency ≤ 1%) and non-synonymous variant observed in 2 or more individuals of the 1000 Genomes Project. In total, DIEP had 8400 pairs of genes in its negative training set, including 7000 pairs from the Random set and 1400 pairs from the DIDA_NI set. Following a different approach, DiGePred used as its final negative training set around 8,400 pairs from genes with variants in sequencing data from unaffected relatives of the Undiagnosed Diseases Network (UDN) individuals [40].

A summary of all the types of features shared and specific to each method can be found in Fig. 1. We considered that a feature was shared between two methods if the same type of information was used as a feature, although the source of the information might be different. Interestingly, two features were included in all the methods: the haploinsufficiency of each gene and the distance between the two genes in a protein-protein interaction (PPI) network. Another feature was present in all methods except VarCoPP: the coexpression of genes in tissues, for instance from the COXPRESdb database [41]. VarCoPP and VarCoPP2.0 also shared variant level features: the CADD scores of the 4 alleles of the pair. Another feature included in VarCoPP was the recessiveness (probability of being a recessive disease gene [42]) of each gene, which was also considered by DIEP. DIEP and DiGePred shared some features: the intolerance to loss-of-function variants of each gene (difference between actual mutation incidence and expected mutation frequency), and the phenotype similarity between the genes. The phenotype similarity between genes has to be distinguished from the phenotypic description of the patient needed in OligoPVP and Hop. Here, the overlap between the phenotype annotations of each gene (from the Human Phenotype Ontology, HPO) is used as a feature in the ML algorithm, but no HPO term has to be provided by the user of the method. DiGePred shared one more feature with VarCoPP2.0: the selection pressure of each gene, computed with the dN/dS ratio (ratio between the non-synonymous mutations and synonymous mutations during evolution). All other features used for training the ML-sympt-agno methods were method-specific.

Fig. 1.

Venn diagram of the features of the random forest methods to detect digenic inheritance.

In the article describing each ML-sympt-agno method, the Gini importance of each feature, which represents its contribution to the prediction (positive or negative), was computed. For VarCoPP, the feature importance was by far the greatest for the CADD score of the first variant allele of each gene, followed by the recessiveness of each of the genes. A similar pattern was observed in VarCoPP2.0 for CADD, but the second most important feature group was gene pair features (PPI distance, coexpression and knowledge graph distance). In DiGePred, the phenotype similarity between genes had the highest importance, followed by pathway similarity, number of phenotypes and PPI distance. For DIEP, the feature with the most importance was by far the PPI distance, followed by protein functional interaction effects, biological distance, phenotype similarity and Gene Ontology (GO) [43] annotations.

ARBOCK (Association Rule learning Based on Overlapping Connections in Knowledge graphs) takes a different approach as compared to the RF-based methods [28]. The ARBOCK method uses an association rule, based on a knowledge graph, which makes inferences by leveraging the structured information stored within the knowledge graph. ARBOCK is based on the BOCK heterogeneous knowledge graph integrating oligogenic disease information, which comprises 158,964 nodes of 10 different types (genes, biological processes, protein families, associated phenotypes, protein domains, diseases, molecular functions, cellular components, protein complexes and oligogenic combinations) and 2,659,064 edges of 17 different types (eg. interaction between gene nodes, association between a gene node and a phenotype node, or similarity between biological process nodes). ARBOCK uses a two-step approach to predict pathogenic gene interactions. First, it generates a set of rules from local patterns in the knowledge graph of the known pathogenic gene pairs. Then, these rules are combined into a decision set classifier to allow the detection of new pathogenic gene pairs. For training, ARBOCK used 441 gene pairs from the latest version of OLIDA with sufficient evidence of pathogenicity, and 44,100 putatively neutral gene pairs from the 1000 Genomes cohort. The importance of features in ARBOCK cannot be compared directly to the other ML methods based on RF algorithms with a score like the Gini importance of each feature. It can still be noted that in ARBOCK, high-confidence rules are characterized by metapaths (sequence of node and edge types) related to similar phenotypes (50%), biological processes (31.2%), and molecular functions (12.5%) which indicates key information considered by the classifier.

Overall, the ML-sympt-agno methods to detect DI all consider information at the gene and gene pair level but with many differences and specificities. Considering the variability of the features, of the positive and negative training sets they take as input, as well as the different ML models, a benchmarking of these methods would be necessary for their in-depth comparison.

Methods requiring patient phenotype as input (ML-sympt-info)

The other individual-based methods to detect DI rely on the phenotypic characterization of the patient in addition to genetic data. Both methods rely on ML scores, but as they take as input other type of data we will consider them separately from purely ML methods to detect digenism. OligoPVP [25] combines a monogenic predictor of variant pathogenicity, the PhenomeNET Variant Predictor (PVP) version 2.0 (“DeepPVP”) [44] with the information from a gene network database, STRING [45]. DeepPVP is a ML score which predicts variant pathogenicity using the following features: pathogenicity predictors (CADD [46], DANN [47] and GWAVA [48]), a phenotype similarity score (variant are prioritized based on their gene being associated with the phenotypes observed in the patient), disease inheritance mode and variant genotype. For each individual, if the genes are connected in the STRING network, OligoPVP sums the PVP scores of their variants to make a score by pair of genes. It also allows the exploration of triplets of variants. The input file is a VCF file and the OMIM [49] ID of the disease or a list of phenotypes (HPO [50] or Mammalian Phenotype Ontology (MPO) [51] terms), which are used to calculate the PVP scores of variants. The strength of this method is its applicability to all variants from a patient’s exome through a user-friendly command line tool. A main limitation is its reliance on HPO terms and its limitation to pairs of genes interacting directly in the STRING database. Indeed, if the genes of the pair containing the tested variants are not connected through the STRING network, the pair is not prioritized. In addition, the method relies on the phenotypic characterization of the patient with the PVP score and thus is biased towards known phenotype-gene or phenotype-disease associations.

The most recent method to detect DI is the High-throughput oligogenic prioritizer (Hop) [34]. Hop leverages both the VarCoPP2.0 predictor and the BOCK knowledge graph mentioned above [28]. The input of the method is a VCF file, along with either a combination of HPO terms describing the patient’s phenotype or a panel of genes known to be involved in the disease or both. The VCF file is filtered to remove variants with a MAF ≥ 0.035, synonymous variants that are further than 195 bp from exon edges and intronic variants. Then, Hop calculates two scores. The first score is calculated using VarCoPP2.0, which evaluates the pathogenicity of all possible variant combinations from the filtered VCF file. The second score is a disease-relevance score for all gene pairs that is calculated using a random-walk with restart algorithm with the provided disease-related terms as seeds on the BOCK heterogeneous knowledge graph. These two scores are then averaged to provide a final score for ranking all of the variant pairs of the individual.

Despite promising results on synthetic digenic disease exomes, Hop is still limited by its reliance on prior knowledge, especially regarding the link between HPO terms and disease genes in the knowledge graph. The Hop method also works better if a relevant gene panel for the disease is provided, which limits the use of the method to well-studied disease.

Benchmark of machine learning algorithms to detect digenic inheritance

Overview of the benchmark

ML-sympt-agno methods do not need as input a cohort of patients (unlike the DM and RareComb) or patient phenotypic information (unlike OligoPVP and Hop) to make predictions. ML-sympt-agno methods are thus suited for the detection of DI in the context of very rare and heterogeneous diseases. All ML-sympt-agno methods output a score between 0 and 1 and provide a threshold above which the pair is classified as pathogenic, which makes them easily comparable. For a comprehensive benchmark of existing methods to find DI in sequencing data, we thus focused on the ML-sympt-agno methods currently published and for which a program or implementation is available. We excluded VarCoPP, which has been replaced VarCoPP2.0 in the ORVAL platform. We hence tested four methods: DiGePred, DIEP, VarCoPP2.0 and ARBOCK. The specific threshold and software parameters used for each method can be found in Supplementary Note 1. For ARBOCK, we tested the two predictive models available to evaluate new pairs of genes for an exhaustive comparison: a model excluding (ARBOCK.excl.pheno strategy) and a model including (ARBOCK.incl.pheno strategy) gene-associated phenotypic information as a training feature of the ML algorithm. All methods except VarCoPP2.0 take as input pairs of genes. VarCoPP2.0 is run through the web platform ORVAL and takes as input a list of variants, creating all possible pairs of variants from the provided list.

The overall aim of this benchmark is to test the classifying performance of the ML-sympt-agno methods under different scenarios, leveraging pairs for which the classification as pathogenic or neutral (non-pathogenic) is known or very likely. We were not able to screen a full individual exome to test all possible gene pairs which would be the most comprehensive approach: the ORVAL platform especially limits the number of input variants. We focused our benchmark to two scenarios that address a range of potential combination occurrences from true datasets without having to explore a whole exome. The full description of all scenarios and the pairs for evaluation can be found in Supplementary Table S1. The precise selection criteria for pairs are described in the Supplementary Materials Notes 2 and 3.

First, we evaluated the performance of ML-sympt-agno methods to detect digenism in a scenario where one of the OLIDA digenic pairs was included in an individual genome (scenario 1). We evaluated scenario 1 by testing if ML-sympt-agno methods classified the known digenic pairs (true positives) as pathogenic and other gene pairs assumed to be neutral (true negatives) as not pathogenic. Second, we tested ML-sympt-agno methods on pairs composed of one monogenic variant causing disease and one neutral variant (true negatives) that should be classified as not pathogenic in the case of DI (scenario 2).

For the positive testing sets, we focused on the 69 digenic pairs from OLIDA with evidence of oligogenicity (FINALmeta score ≥ 1) that were not used to train VarCoPP2.0 and that did not involve CNVs (OLIDA pairs). Indeed, CNVs cannot be scored with VarCoPP2.0. The OLIDA pairs included 14 gene pairs that were specifically not used to train the ARBOCK method (ARBOCK_digenic pairs, Supplementary Note 2), which we also tested separately as some of the OLIDA pairs were used to train ARBOCK. Both of these positive testing sets were used in the case of benchmarking scenario 1.

For the negative testing sets, we leveraged variants observed in 10 individuals randomly selected from the FREX [52] project to avoid bias from selecting variants from a single individual. Each neutral pair of genes was created either by selecting randomly two variants in a FREX individual (random_FREX pairs, N = 32,953 pairs) or by selecting two variants among the 100 variants prioritized as most pathogenic in the individual by Easy-PSAP [53] (https://github.com/msogloblinsky/Easy-PSAP, prioritized_FREX pairs, N = 18,348 pairs). The latter case corresponds to a filtering step that would involve shortlisting the variants with the highest predicted pathogenicity in an individual and searching for DI within that shortlist. Indeed, variants involved in a pathogenic gene pair are expected to have an impact on the function of the gene, which can be evaluated through pathogenicity scores. Selecting variants predicted as most pathogenic in an individual can challenge methods like VarCoPP2.0 which use variant level data like the CADD score for prediction. We also shuffled the genes implicated in the ARBOCK_digenic pairs except the initial true digenic pairs (ARBOCK_shuffled pairs), to test performance of the ML-sympt-agno methods specifically on genes already involved in digenic pairs. This set of pairs was assumed to be mostly negative but we cannot exclude that a few of them are truly digenic. The random_FREX, prioritized_FREX pairs and ARBOCK_shuffled pairs were used as negative testing sets for benchmarking scenario 1. Finally, we used a curated list of well-reviewed pathogenic variants [53] from the ClinVar database [35] associated with a known autosomal dominant (AD) or autosomal recessive (AR) mode of inheritance (100 variants from 100 different genes for the AD and AR models, respectively). This led to four additional negative testing sets involving a monogenic variant from ClinVar (with AD or AR inheritance) and a neutral variant from either of the previously described two sets of variants from FREX, to test benchmarking scenario 2: random_FREX_clinvar_AD, random_FREX_clinvar_AR, prioritized_FREX_clinvar_AD and prioritized_FREX_clinvar_AR (Supplementary Note 3). Because the prioritized FREX variants are possibly pathogenic, combining them with the well-reviewed pathogenic variants from the ClinVar database to build pairs can lead to a few pairs being truly digenic, but we assumed the large majority of the pairs are not digenic.

Benchmarking scenario 1: performance for detecting digenism

Classification of digenic gene pairs from OLIDA compared to neutral pairs from FREX

Scenario 1 of this benchmark aimed at evaluating the performance of the ML-sympt-agno methods to detect a digenic pair compared to neutral pairs seen in the general population, either taken randomly or among the most predicted pathogenic variants of the individual. We first calculated evaluation metrics for each ML-sympt-agno method (Table 3, Supplementary Table S2), by using OLIDA pairs (N = 69) as true positives and either all of FREX pairs (all, N = 51,301), only random_FREX pairs (random, N = 32,953) or only prioritized_FREX pairs (prioritized, N = 18,348) as true negatives. Considering the minimal differences in evaluation metrics when using the “all”, “random” or “prioritized” as true negatives, we only show here the results of the evaluation with “all” true negative FREX pairs pooled together. Considering the Positive Predictive Value (PPV), Negative Predictive Value (NPV) and F1 score are all sensitive to class imbalance, we calculated these metrics, as well as the specificity, for the 69 positive pairs and 69 randomly downsampled negative pairs, and replicated the process 10,000 times (Supplementary Fig. S1, Table 3, Supplementary Table S2).

Table 3.

Evaluation metrics for all benchmarked methods to detect DI.

Method	Sensitivity	Median specificity	Median PPV	Median NPV	Median F1 score
ARBOCK.excl.pheno	0.870	0.696	0.741	0.842	0.800
ARBOCK.incl.pheno	0.928	0.928	0.928	0.928	0.928
DIEP	0.855	0.986	0.983	0.872	0.915
DiGePred	0.638	1	1	0.734	0.779
VarCoPP2.0	0.493	1	1	0.663	0.660

For metrics considering the negative training set (all metrics except sensitivity), median values are shown for 10,000 replicates of downsampling the negative testing set to have an equal number of pairs as in the positive testing set; sensitivity: proportion of true digenic pairs classified as pathogenic, specificity proportion of neutral pairs classified as not pathogenic, PPV positive predictive value: proportion of pairs classified as pathogenic that are true digenic pairs, NPV negative predictive value: proportion of pairs classified as not pathogenic that are neutral pairs, F1 score harmonic mean of precision and recall.

The method with the highest sensitivity and NPV is ARBOCK.incl.pheno, which has overall better metrics values than ARBOCK excl.pheno. ARBOCK.incl.pheno also achieves the highest F1 score, which measures the predictive performance of a classifier, followed by DIEP. The F1 score is the harmonic mean of precision, the ratio of true positive predictions to the total number of positive predictions made by the model, and recall, the ratio of true positive predictions to the total number of actual positive instances. The highest specificity and PPV were achieved by DiGePred and VarCoPP2.0. Between the two methods, DiGePred had the highest sensitivity and F1 score. Finally, ARBOCK.excl.pheno did not achieve a high score for any of the evaluation metrics (e.g. all values < 0.9) although its median F1 score still reached 0.8.

When looking more precisely at pairs in each category, all ML-sympt-agno methods classified around half to almost all of the OLIDA pairs as pathogenic (Fig. 2, Supplementary Fig. S2, Supplementary Table S3). The best classification of true positives is achieved by ARBOCK.incl.pheno (64 well-classified of 69 positive pairs) and then ARBOCK.excl.pheno and DIEP with 60 and 59 well-classified positive pairs, respectively. The results were more contrasted between methods for the FREX neutral pairs. For the random_FREX pairs, DiGePred and VarCoPP2.0 classified as pathogenic only 16 and 10 out of 32,953 pairs, respectively. In contrast, ARBOCK.excl.pheno classified 30% of random_FREX pairs as pathogenic (10,056 misclassified negative pairs). ARBOCK.incl.pheno wrongly classified 2525 of the random_FREX pairs (7.6%), whilst DIEP misclassified only 2.5% of them. This trend of classification is found again for the prioritized_FREX pairs, with DiGePred keeping the lowest number of misclassified pairs. The only main difference in classification trend between the two negative testing sets was found for VarCoPP2.0, for which the percentage of misclassified pairs increased from 0.03% for the random_FREX pairs to 2.4% for the prioritized_FREX pairs.

Fig. 2. Distribution of ML-sympt-agno methods scores for the OLIDA pairs and the two scenarios with FREX variant pairs.

Method-specific quantitative threshold (grey) recommended for declaring a pair as pathogenic; the line splitting the boxplot in two represents the median value.

Classification of digenic gene pairs held-out from ARBOCK method

To corroborate the conclusions of benchmarking scenario 1, we tested the ML-sympt-agno methods on the 14 ARBOCK_digenic (digenic pairs specifically not used to train any of the ML-sympt-agno methods including ARBOCK) and the 364 ARBOCK_shuffled pairs (not digenic pairs involving the same genes as ARBOCK_digenic pairs). Overall, DiGePred and VarCoPP2.0 had the worst performance in terms of classifying the ARBOCK_digenic pairs as pathogenic (4 and 2 pairs classified as pathogenic by DiGePred and VarCoPP2.0, respectively; Fig. S3, Supplementary Table S4). The best results on ARBOCK_digenic pairs were achieved by ARBOCK: ARBOCK.excl.pheno and ARBOCK.incl.pheno classified 11 and 9 pairs as pathogenic, respectively. However, when looking at the ARBOCK_shuffled negative testing set, ARBOCK exhibits an extremely high misclassification rate. In contrast, DiGePred has the lowest rate of misclassification of the negative pairs (2%), followed by DIEP (12%) and VarCoPP2.0 (16%).

Benchmarking scenario 2: performance for differentiating the monogenic from the digenic model

A final aspect that we included in this benchmark is scenario 2, which evaluates the ability of ML-sympt-agno methods to distinguish between DI and monogenic inheritance. In this scenario, we had 4 negative testing sets: random_FREX_clinvar_AD, random_FREX_clinvar_AR, prioritized_FREX_clinvar_AD and prioritized_FREX_clinvar_AR. For all of the sets, DiGePred did not classify any of the pairs as pathogenic (Fig. 3, Supplementary Fig. S4, Supplementary Tables S5 and S6). DIEP and VarCoPP2.0 gave almost similar results, with less than 10 pairs wrongly classified for any scenario. VarCoPP2.0 performed slightly worse for the two sets using prioritized_FREX compared to random_FREX, going from 2 to 6 wrongly classified pairs for the AD model and from 1 to 7 wrongly classified pairs for the AR model. The worst classification performances were observed for ARBOCK.excl.pheno for the scenarios involving ClinVar AD variants, with 32 pairs (random_FREX_clinvar_AD) or 36 pairs (prioritized_FREX_clinvar_AD) classified as pathogenic. ARBOCK.incl.pheno performed better than ARBOCK.exl.pheno but worse than all of the other methods.

Fig. 3. Distribution of ML-sympt-agno methods scores for the two scenarios involving ClinVar and random FREX variant pairs.

Method-specific quantitative threshold (grey) recommended for declaring a pair as pathogenic; the line splitting the boxplot in two represents the median value.

Discussion

Several methods have been developed recently to tackle the complex issue of detecting DI in sequencing data, but this remains an under-explored field of research. Indeed, several of these methods have been developed by the same team (VarCoPP, VarCoPP2.0, ARBOCK and Hop) which also curated the DIDA and OLIDA databases. Our benchmark highlights that cohort-based methods to detect DI are largely applicable to sequencing data but need a substantial number of individuals, and so are not suited for rare diseases. Cohort-based methods also require a strong prefiltering step of variants to ensure adequate power. Despite exhaustively evaluating all of the possible variant or gene pairs still being very computationally intensive and thus not practically feasible with existing individual-based methods to detect DI except VarCoPP2.0, they can in theory be applied to any pair of genes or variants. Although it is difficult to pinpoint which level of information contributes the most to accurate predictions of digenism, the key predictive features described for each ML-sympt-agno method seem to indicate that gene-level and gene-pair level features play a key role in predicting DI. Regarding methods taking as input the phenotypic description of a patient, OligoPVP requires the presence of known protein-protein interaction between genes which limits the analysis. Both OligoPVP and Hop rely on phenotypic data to make an inference, which can be limited if it is not available or not precise for the patient.

A significant issue in the application of the ML-sympt-agno methods to detect DI is their false positive rate. DiGePred showed the highest specificity in our evaluations. A downside of DiGePred is that phenotype-related gene features play the most important role in DiGePred predictions, as described in the DiGePred article. This reliance on phenotype annotation of genes as a feature can hinder the generalization of the model when these features are missing or incomplete for a gene. That is evident from the ARBOCK article and confirmed by our benchmark: DiGePred was only able to predict correctly 4 out of 14 ARBOCK_digenic pairs, as compared to 11 out of 14 ARBOCK_digenic pairs for ARBOCK.excl.pheno, which was not trained using phenotype-based features. ARBOCK.incl.pheno did not suffer from the same generalization issue as DiGePred, although it performed less well than ARBOCK.excl.pheno by classifying 9 out of the 14 ARBOCK_digenic pairs as pathogenic. Therefore, DiGePred is a more conservative method that tends to fail to predict gene pairs as pathogenic if they lack complete phenotype annotation or have limited common phenotype terms. DIEP was described as a better predictor of DI than DiGePred, as it included a larger and more finely filtered negative training set and broader range of features. Indeed, DIEP predicts more of the OLIDA (59 for DIEP as compared to 44 for DiGePred) and ARBOCK_digenic pairs (8 for DIEP against 4 for DiGePred) as pathogenic. However, DIEP also produces a lot more false positive predictions across all the negative testing sets. In contrast to DiGePred, the most important features in DIEP are protein–protein association, functional interaction, and semantic similarity of gene GO annotations. The two ARBOCK predictive models have both the highest sensitivity and lowest specificity. This behavior of the method had already been described in the article by [25], which states that the ARBOCK pathogenicity predictor excluding phenotypic information, exhibits a relatively high false positive rate. In our benchmark, even when ARBOCK included phenotypic information, the number of false positives was still very high compared to the other methods, which can partly be explained by the uncertainty surrounding the selection of neutral gene pairs to train the method. It can also be noted that only DiGePred did not misclassify any of the pathogenic ClinVar variant and neutral FREX variant pairs

One observation already highlighted in the DiGePred article when comparing DiGePred to the initial VarCoPP model is that VarCoPP’s predictions were strongly influenced by variant-level features, especially if one of the genes carries a variant predicted to be highly pathogenic, which may hide or confound purely digenic effects. This behavior could also be seen in our benchmark, as VarCoPP2.0 wrongly classified as pathogenic only 10 pairs of the 32,953 neutral random_FREX pairs and 440 pairs of the 18,348 neutral prioritized_FREX pairs that include, for each individual, variants prioritized among the most pathogenic. This observation was expected since the most important feature of VarCoPP2.0 is the CADD scores of the variants, and the variants in the prioritized_FREX pairs have high CADD scores by construction. Still, the VarCoPP2.0 model was enriched in gene-pair level features compared to VarCoPP, which improved the capture of the gene-gene synergistic relationship inside a variant pair that is crucial for DI prediction. In the article describing VarCoPP2.0, the authors highlight that the method has a high sensitivity, which is important in a clinical setting, as well as a 5% false positive rate, and should be used in conjunction with variant filtering and/or restriction to a panel of genes. The currently advised variant filtering for VarCoPP2.0 in ORVAL is to remove variants with a MAF ≥ 0.035, intergenic, intronic and exonic synonymous variants. We did not select variants according to these criteria as it would have reduced even more our positive and negative testing sets (64 pairs in the OLIDA set, 160 pairs in the random_FREX set and 2012 pairs in the prioritized_FREX set) but following this approach could have improved VarCoPP predictions. VarCoPP also presents the interest of being able to supply a list of genes relevant to the disease of interest to further narrow the number of examined combinations.

Overall, our main recommendation for potential users of ML-sympt-agno methods to detect DI is to use the ARBOCK including phenotype feature information method if the aim is to have the best prediction performance. If the aim is to limit the number of false positives, in the case of the exploration of a high number of gene pairs for instance, DiGePred will be better suited. Finally, a method that stood out as striking a good balance between true and false positive is DIEP, which achieved a similar F1 score to ARBOCK.

A main limitation of our benchmark, and of all the aforementioned benchmarked methods, is the small number of known digenic pairs used as positive training and testing sets. The OLIDA database is updated regularly, and the performance of the methods to detect DI will have to be tested on new pairs described in the literature. Our positive testing set can also be biased against specific disease and genes that are overly represented in OLIDA due to their known digenic or oligogenic nature. Regarding the applicability of the benchmarked methods, both DiGePred and DIEP provide pre-calculated tables of prediction for all human gene pairs. DIEP even presents the advantage of providing a high-efficiency java package to search for specific digenic pairs or genes. These two methods, as well as ARBOCK, present the advantage of separating the selection and effect of variants from the identification of potentially digenic gene pairs. Indeed, the question of variant impact has been extensively explored by different methods, whereas reflecting the potential impact of a simultaneous alteration of two genes is very specific to digenism. VarCoPP2.0 is not suitable for whole exome analysis, especially due to the challenge of running the web server on a large scale.

ML-sympt-agno methods to detect DI are not yet at a stage allowing an agnostic exploration of real-world cases of undiagnosed diseases but can serve as valuable information combined with other lines of evidence like familial data if available. They also lack interpretability in their predictions. ORVAL and ARBOCK tried to mitigate the issue of interpretability by offering more context at the gene and gene pair level such as visual mappings of the cellular location and pathway information and explanation in the form of subgraphs, respectively. Methods continue to be published on the subject of DI detection regularly, and are either individual-based [54] or cohort-based [31, 55]. Overall, the appropriate integration of variant selection and gene pair prediction will in the future lead to more accurate prediction of DI, and facilitate the detection of this complex mode of inheritance. Finally, with the availability of different resources providing sequencing data from large population samples such as gnomAD [56], UKBioBank [57] or AllOfUs [58], it is now possible to determine if a given pair of variants in two different genes is present or not in healthy people. This information could be used to train machine learning models as was done to estimate the pathogenicity of single variants. Following a different approach, it could also lead to the extension of methods such as the PSAP method [59] and PSAP-genomic-regions [53], which could leverage these resources and the general population as controls to estimate the likelihood for a combination of variants seen in a given patient to be disease causative.

Author contributions

M-S.O., E.G., G.M. and D.C. contributed to the study conception and design. The FrEx Consortium provided data. The literature review and benchmark analysis were performed by M-S.O. The first draft of the manuscript was written by M-S.O. E.G., G.M., A.B., E.T-L. and D.C. commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

This work was jointly supported by the French Priority Research Program on Rare Diseases “Programme Prioritaire de Recherche Maladies Rares du Programme français d’Investissements d’Avenir” and by the Brittany region through the ARED program to E. Génin, and by the National Institutes of Health of the United States of America grants R01HD078641 and P50HD096723 to D. F. Conrad. This work was also supported by the INSERM GOLD Cross-Cutting program.

Data availability

This study used individual exome data from the FREX project generated by the France Génomique National infrastructure that can be contacted for data access at https://www.france-genomique.org/databases/frex-the-french-exome-project-database.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41431-025-01834-9.