NIAPU: network-informed adaptive positive-unlabeled learning for disease gene identification

Abstract

Motivation

Gene–disease associations are fundamental for understanding disease etiology and developing effective interventions and treatments. Identifying genes not yet associated with a disease due to a lack of studies is a challenging task in which prioritization based on prior knowledge is an important element. The computational search for new candidate disease genes may be eased by positive-unlabeled learning, the machine learning (ML) setting in which only a subset of instances are labeled as positive while the rest of the dataset is unlabeled. In this work, we propose a set of effective network-based features to be used in a novel Markov diffusion-based multi-class labeling strategy for putative disease gene discovery.

Results

The performances of the new labeling algorithm and the effectiveness of the proposed features have been tested on 10 different disease datasets using three ML algorithms. The new features have been compared against classical topological and functional/ontological features and a set of network- and biological-derived features already used in gene discovery tasks. The predictive power of the integrated methodology in searching for new disease genes has been found to be competitive against state-of-the-art algorithms.

Availability and implementation

The source code of NIAPU can be accessed at https://github.com/AndMastro/NIAPU. The source data used in this study are available online on the respective websites.

Supplementary information

Supplementary data are available at Bioinformatics online.

1 Introduction

The discovery of gene–disease associations (GDAs) is made difficult by incomplete knowledge of biological and physiological processes. When approaching complex, multi-gene diseases and traits, it is hard to disentangle the contribution of each gene, and computational biological approaches for predicting GDAs (Opap and Mulder, 2017; Piro and Cunto, 2012) can support and address experimental methods (e.g. genome-wide association studies—GWAS—or linkage studies, among others) which are expensive and time-consuming.

The fuzzy background of yet unknown or truly unassociated genes contributes to making the computational identification of disease genes challenging to carry out with accuracy. In machine learning (ML), this setting translates into the ability to identify new positive instances among a set of positive and unlabeled samples, a task known as ‘positive-unlabeled (PU) learning’ (Bekker and Davis, 2020; Liu et al., 2003). This task can be addressed through semi-supervised learning algorithms, trained using two approaches. In the first one, the set of unlabeled instances is assumed to be a contaminated set of negative instances and the contamination is considered during the modeling process by weighting the data points or adding penalties on misclassification (Claesen et al., 2015; Elkan and Noto, 2008; Ke et al., 2018; Mordelet and Vert, 2014). In the specific case of gene discovery, this contamination is given by the possibility of the negative instances of containing not yet discovered positive genes. The second approach, called two-step technique, aims at relabeling the instances and then training a supervised learning algorithm (Liu et al., 2003; Yang et al., 2012, 2014). For example, Yang et al. (2012) introduced a multi-class labeling procedure considering five different labels, namely Positive (P), Likely Positive (LP), Weakly Negative (WN), Likely Negative (LN) and Reliable Negative (RN), based on a Markov process with restart (Can et al., 2005), widely applied in disease genes identification (Köhler et al., 2008; Li and Patra, 2010a, b). Then, a supervised learning algorithm is trained on the relabeled data.

In the present work, we considered the multi-class labeling approach since it allows identifying a set of originally unlabeled items, namely the LP set, whose features are close to that of the items in P. This translates into the identification of a small set of genes more likely to contain true positive instances, hence providing a set of new candidate disease genes for prioritization.

Going beyond the approach from Yang et al. (2012), we propose several significant modifications of the multi-class method regarding the distance matrix defining the Markov process and the selection of the different classes. Some of these modifications were needed in order to apply the method to general PU datasets, while others were proposed to make the process of class formation more rigorous and, at the same time, flexible. The approach considered here, being a two-step technique, is based on the separability and smoothness assumptions (Bekker and Davis, 2020), which require that the features should be able to distinguish between positive and negative instances and, at the same time, instances with similar features should be more likely to have the same label. Therefore, as a further contribution, we propose the use of specific network-informed features, one of them introduced for the first time in this work, based on protein–protein interaction (PPI) data, which provide a characterization of the topological relationships of all human genes with respect to the set of disease genes. The use of such measures grants a much more precise classification of genes than other topological measures. In particular, the set of seed genes is identified very precisely as well as the genes closest and farthest to them, as shown in Section 3.1. The network-informed adaptive PU (NIAPU) framework is therefore formed by two components: the network diffusion and biology-informed topological (NeDBIT) features and the adaptive PU (APU) labeling algorithm.

2 Materials and methods

2.1 Data sources and preprocessing

The proposed methodology exploits two types of data, that is, reliable PPIs and known GDA data. PPI data provide valuable biological knowledge for the identification of undiscovered disease genes (Doncheva et al., 2012; Petti et al., 2021; Piro and Cunto, 2012; Silverman et al., 2020; Tieri et al., 2019). Human PPI data, that is, the human interactome, were gathered from the BioGRID (Stark et al., 2006) dataset (version 4.4.206). The human interactome is obtained by choosing Homo sapiens genes (organism ID 9606), from which we extract a connected network consisting of 19 761 genes and 678 932 non-redundant, undirected interactions (see Supplementary File S1).

GDAs were derived from DisGeNET (version 7.0) (Piñero et al., 2016, 2020), a discovery platform containing one of the largest publicly available collections of genes and variants associated with human diseases together with a score denoting the association confidence and significance. Ten diseases were considered: malignant neoplasm of breast (disease ID C0006142, 1074 genes), schizophrenia (C0036341, 883 genes), liver cirrhosis (C0023893, 774 genes), colorectal carcinoma (C0009402, 702 genes), malignant neoplasm of prostate (C0376358, 616 genes), bipolar disorder (C0005586, 477 genes), intellectual disability (C3714756, 447 genes), drug-induced liver disease (C0860207, 404 genes), depressive disorder (C0011581, 289 genes) and chronic alcoholic intoxication (C0001973, 268 genes). The selection criterion for these diseases was the highest cardinality of GDAs in the curated DisGeNET dataset to ensure sufficient information for the ML task. To validate the gene discovery results, we relied on the all genes DisGeNET dataset, which we refer to as extended dataset. The latter contains associated genes from additional sources not present in the curated version (Bravo et al., 2014, 2015; Bundschus et al., 2008, 2010). More details can be found in Supplementary File S2. After performing additional cleaning steps (see Supplementary File S2), we ended up having a set of seed genes for each disease, denoted by Σ, with their association score S. In particular, we have 1025 genes for disease C0006142, 832 for C0036341, 747 for C0023893, 672 for C0009402, 606 for C0376358, 451 for C0005586, 431 for C3714756, 320 for C0860207, 279 for C0011581 and 255 for C0001973.

2.2 Multi-class labeling: APU labeling algorithm and classification

The APU algorithm consists of a multi-class labeling procedure that relies on the labels introduced in Yang et al. (2012): P, LP, WN, LN and RN. P instances are the known disease genes, RN instances represent the genes whose features are the furthest from the average features in the P set, while the remaining labels are assigned through a Markov process with restart (Can et al., 2005). The novelty of the proposed method is the construction of a new transition matrix starting from the distance matrix between the features of the genes. The matrix needs to be normalized in order to preserve the total transition probability of the state vector whose initial value is different from zero only for genes in the P and RN classes. Moreover, the class selection has been made flexible by using an adaptable quantile separation instead of fixed thresholds. These characteristics have been implemented in order to make the process of class formation more rigorous and, at the same time, more flexible hence easily adaptable to different settings, datasets and needs.

Let V be a set whose generic ith element $v_{i=1,\dots ,n}$ is characterized by the couple $\left({\mathit{x}}_i,y_i\right)$ where ${\mathit{x}}_i\in {\left[0,1\right]}^d$ represent the feature vector, and $y_i\in \left\{0,1\right\}$ the initial label. The APU algorithm is defined by the following steps:

Step 1 : Compute the matrix W, whose elements w_ij are defined as follows:

$$w_{ij}=\{\begin{array}{cc}1-\frac{e_{ij}-m}{M-m}\hfill & \text{if} \hspace{0.17em}i\ne j\hfill \\ 1\hfill & \text{otherwise}\hfill \end{array},$$ (1)

where $e_{ij}=\sum _k{\left(x_i^k-x_j^k\right)}^2,\hspace{0.17em}m={\text{min}}_{ij}\left\{e_{ij}\right\}$ and $M={\text{max}}_{ij}\left\{e_{ij}\right\}$. The symmetric matrix W represents the similarity score between elements i and j.

Step 2 : Compute the reduced matrix ${\mathit{W}}_r$ as follows:

$$w_{r,ij}=\{\begin{array}{cc}{w}_{ij}\hfill & \text{if} \hspace{0.17em}{w}_{ij}>q_{\mathit{w}}\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array}.$$

The threshold $q_{\mathit{w}}$ is computed as a given quantile of the distribution of the elements in the matrix W in order to exclude from the propagation process links between poorly related elements. To obtain a proper Markov process, that is, preserving the probability distribution, the matrix ${\mathit{W}}_r$ must be normalized as ${\mathit{W}}_n={\mathit{D}}^{-1}{\mathit{W}}_r$, where D is the diagonal matrix with elements $d_{ii}=\sum _j{w}_{r,ij}$.

Step 3 : Initialize the propagation process with the initial state vector ${\mathit{g}}_0$ defined as follows. Let $\left|P\right|$ be the cardinality of P (set of seed genes) and $\widehat{\mathit{x}}=\left({\widehat{x}}^1,\dots ,{\widehat{x}}^d\right)$, where ${\widehat{x}}^k=1/\left|P\right|\sum _{i\in P}{x}_i^k$, be the average features of P. The RN genes are chosen to be the ones having the most distant features from $\widehat{\mathit{x}}$. We select the $\left|P\right|$ most distant genes from $\widehat{\mathit{x}}$ in order to keep the classes balanced. Then, the ith element of ${\mathit{g}}_0$ is defined as

$$g_{0,i}=\{\begin{array}{c}1\hspace{1em}\text{if}\hspace{1em}i\in \mathrm{P}\\ -1\hspace{1em}\text{if}\hspace{1em}i\in \text{RN}\\ 0\hspace{1em}\text{otherwise}\end{array}.$$

When needed, a different number of RN genes can be selected. In this case, the initial value of the RN genes in the state vector ${\mathit{g}}_0$ must be set to $-\left|\mathrm{P}\right|/\left|\text{RN}\right|$ so that the two distributions of positive and negative values are balanced in ${\mathit{g}}_0$, with the sum of its elements equal to zero.

Step 4 : Define a Markov process with restart as

$${\mathit{g}}_r=\left(1-\alpha \right){\mathit{W}}_{\mathit{n}}^{\mathit{t}}{\mathit{g}}_{r-1}+\alpha {\mathit{g}}_0,$$ (2)

where the parameter α is usually set to 0.8 (Li and Patra, 2010a; Yang et al., 2012). Starting from the state vector ${\mathit{g}}_0$, the dynamics in Equation (2) ends in the stationary state ${\mathit{g}}_{\infty }$, numerically reached when $|{\mathit{g}}_r-{\mathit{g}}_{r-1}|<{10}^{-6}$.

Step 5 : Use $G_{\infty }$ to assign the remaining labels. Selecting only the elements that belong neither to P nor to RN, the values of the asymptotic distribution of those elements are sorted and the ranking of the corresponding elements is used to form the remaining classes: LP, WN and LN. A simple rule is to divide the ranking into three equal parts and identify LP samples with the first third, WN with the second third and LN with the third third. However, depending on the type of analysis and the problem addressed, any division of the ranking can be considered acceptable.

Step 6 : Classification. An ML classifier is trained over the dataset containing features and labels. Three different ML algorithms have been used: Random forest (RF) (Breiman, 2001), support vector machine (SVM) (Cortes and Vapnik, 1995; Drucker et al., 1997) and multilayer perceptron (MLP) (Hastie et al., 2001) (details in Supplementary File S2).

2.3 NeDBIT features

The NeDBIT features include two network diffusion-based features, namely heat diffusion and balanced diffusion, and two biology-informed topological metrics, namely NetShort and NetRing. Network diffusion methods are widely used in disease gene discovery processes (Janyasupab et al., 2021; Lancour et al., 2018; Picart-Armada et al., 2019). We coupled network diffusion methods and innovative topological-based features in order to make the most of the combined predictive power of both approaches. Moreover, all the features are computed exploiting the association score S. In this way, the NeDBIT features, not assigning the same weight to all seed genes, are certainly more significant for the disease under investigation.

2.3.1. Heat diffusion feature

This feature is obtained by using a heat diffusion process over the network, which is among the most used processes for disease gene prioritization and prediction [see Carlin et al. (2017) and references therein]. Starting with a distribution of weights, with positive values only on the seed genes, their evolution is determined by using the diffusion equation on graph (Nitsch et al., 2010)

$${\mathbf{z}}^{\prime }(t)+\mathbf{L}\mathbf{z}(t)=0,$$ (3)

where $\mathbf{L}$ is the Graph Laplacian matrix, $\mathbf{L}=\mathbf{K}-\mathbf{A}$, $\mathbf{K}$ is the diagonal matrix with the degree of nodes on the diagonal, namely ${\mathbf{K}}_{ii}=k_i$ and $\mathbf{A}$ is the adjacency matrix of the PPI. The weights at time t are given by the formal solution of Equation (3)

$$\mathbf{z}(t)=\text{exp}\hspace{0.17em}\left(-\mathbf{L}t\right)\mathbf{z}(0),$$ (4)

where $\text{exp}\hspace{0.17em}$ is the exponential of the matrix. Regarding the initial distribution of weights, we assign $z_i(0)=s_i$ for seed genes in Σ and 0 otherwise, where s_i is the association score.

2.3.2. Balanced diffusion feature

This feature is obtained by using the diffusion equation in (3) but with another version for the Graph Laplacian matrix, that is, ${\mathbf{L}}_b=\mathbf{I}-{\mathbf{K}}^{-1}\mathbf{A}$. The weights at time t are obtained as in Equation (4) by using operator ${\mathbf{L}}_b$ and the initial weights are given as for the previous measure.

This form of the graph diffusion operator differs from the heat diffusion in the fact that the operator $\mathbf{L}$ diffuses the same amount of score for each link, whereas ${\mathbf{L}}_b$ diffuses the same amount of score for each node. This implies a different short-time behavior of the diffusion process on the graph.

2.3.3. NetShort

The NetShort measure (White and Smyth, 2003) is based on the idea that a generic node is topologically important for a disease if a large number of seed nodes must be traversed to reach it. For each node, the weights are assigned as follows:

$$w_{ij}=a_{ij}\frac{2}{{\tilde{s}}_i+{\tilde{s}}_j},\hspace{1em}\text{where}\hspace{1em}{\tilde{s}}_i=\{\begin{array}{c}\frac{s_i}{\max \mathbf{S}}\hspace{1em}\text{if}\hspace{1em}i\in \Sigma \\ \alpha \frac{\min \mathbf{S}}{\max \mathbf{S}}\hspace{1em}\text{if}\hspace{1em}i\notin \Sigma \end{array}$$

and $\min \mathbf{S}$ and $\max \mathbf{S}$ are the minimum and the maximum of the association scores, α is the penalization parameter given to non-seed nodes and a_ij is the (i, j) element of the adjacency matrix $\mathbf{A}$. We use $\alpha =0.5$ so that all non-seed nodes have normalized score ${\tilde{s}}_i=\frac{1}{2}\frac{\min \mathbf{S}}{\max \mathbf{S}}$ while seed nodes have normalized score $\frac{\min \mathbf{S}}{\max \mathbf{S}}\le {\tilde{s}}_i\le 1$. Then, the NetShort measure NS_i of node i is defined as

$${\text{NS}}_i=\sum _{j\ne i}\frac{1}{d_{ij}},$$

where d_ij is the length of the weighted shortest path from i to j.

2.3.4. NetRing

The NetRing measure, introduced for the first time in this work, is based on the concept of ring structure (Baronchelli and Loreto, 2006) generalized to a set of seed nodes. Starting from seed nodes, a partition of the graph in sub-graphs, or rings, is introduced with the following property:

$$R(l)\equiv \left\{j\in V|\underset{i\in \Sigma }{\min }{l}_{ij}=l\right\},$$

where l_ij is the (unweighted) length of the shortest path from i to j. R(l) contains all the non-seed nodes with a minimal distance l from, at least, one seed node. From the definition follows that $R(0)\equiv \Sigma ,\hspace{0.17em}R(l_1)\cap R(l_2)=\varnothing$ if $l_1\ne l_2$ and $V={\cup }_{l=0}^{L}R(l)$, where L is the highest value of the minimal distance from non-seed nodes to seed nodes.

An initial rank defined by means of the association score is computed as

$${\widehat{r}}_i=\{\begin{array}{c}1-\frac{s_i}{\max \mathbf{S}}\hspace{1em}\hspace{1em}\text{if}\hspace{1em}\hspace{1em}i\in \Sigma \hfill \\ 1\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{if}\hspace{1em}\hspace{1em}i\notin \Sigma \hfill \end{array},$$

then the NetRing measure r_i of node i is defined as

$$r_i=\{\begin{array}{c}\alpha {\widehat{r}}_i+(1-\alpha )\frac{1}{k_i}\sum _{j|A_{ij}\ne 0}{\widehat{r}}_j\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \hspace{1em}\text{if}\hspace{1em}i\in \Sigma \hfill \\ l_i+\frac{1}{k_i}\left(\sum _{j\in O_i}{\widehat{r}}_j+\sum _{j\in R_i(l_i-1)}{r}_j-(l_i-1)\right)\hspace{0.17em}\hspace{1em}\text{if}\hspace{1em}i\notin \Sigma \hfill \end{array},$$

where the score for seed genes is a convex combination of the initial rank ${\widehat{r}}_i$ and the average of the initial rank of the neighbors of the node, so that seed nodes having many seed nodes as neighbors have a higher rank. The rank of non-seed nodes is obtained by summing the level of the ring and the average of two terms, that is, the number of genes belonging to the same or higher rings ($O_i=\left\{j\notin R(l-1)|A_{ij}\ne 0\right\}$) and the sum of the rank of genes in the lower ring ($R_i(l_i-1)=\left\{j\in R(l_i-1)|A_{ij}\ne 0\right\}$) corrected by the ring level. The correction is introduced to make the rank r_j comparable with ${\widehat{r}}_j$. Additional important considerations about the NetRing measure can be found in Supplementary File S2.

3 Results

The performance of NIAPU is tested on the 10 disease datasets detailed in Section 2.1. A visual overview of the workflow can be grasped in Figure 1. Section 3.1 is devoted to testing the performance of NIAPU (APU+NeDBIT) against the implementation of the APU labeling algorithm with two different sets of features commonly used when dealing with disease gene identification. The performances are investigated in terms of out-of-sample classification. Section 3.2 analyzes the performance of NIAPU in the identification of candidate disease genes. To this end, a subset of seed genes is masked out to see whether such genes are predicted as LP. Section 3.3 deals with comparing NIAPU with other disease gene identification algorithms, while Section 3.4 presents results from an enrichment analysis of the candidate disease genes obtained by the NIAPU methodology.

Fig. 1.

The complete NIAPU pipeline. PPI and GDAs are used to obtain a disease-related network. Features are extracted (Section 2.3) and APU is applied (Section 2.2) to assign new labels to train ML algorithms for the final gene classification. The new labels can be used for disease gene-discovery purposes (Sections 3.2 and 3.3).

3.1 NeDBIT classification performances

The effectiveness of the NeDBIT features is tested by comparing NIAPU against the implementation of the APU labeling algorithm with two different sets of features: the first (PUDI) computed following Yang et al. (2012) is based on topological features (originally taken from Xu and Li, 2006) and functional information based on the semantic similarity of GO terms (originally taken from Wang et al., 2007), the second (TFO) includes simple topological, functional and ontological features (see Supplementary Files S2 and S3). The comparison is carried out in terms of out-of-sample classification performance, namely the 10 datasets detailed in Section 2.1 were split into training set (70%) and test set (30%), keeping class balance. Then, we trained the three ML algorithms defined in Step 6 of Section 2.2 for the three different applications of the APU algorithm.

Results related to malignant neoplasm of breast disease are reported in Figure 2 in terms of confusion matrices. The comparison among TFO, PUDI and NeDBIT features shows that the latter are far superior to the others. The joint usage of APU and NeDBIT features (NIAPU) succeeded in discriminating the class P from the rest of the genes and better separating the pseudo-classes LP, WN, LN and RN.

Fig. 2.

Confusion matrices for multi-class classification on malignant neoplasm of breast (C0006142). The APU labeling and the newly defined NeDBIT features allow for a better and clear distinction of the P class and the pseudo-classes. (a) MLP + TFO features. (b) MLP + PUDI features. (c) MLP + NeDBIT features. (d) RF + TFO features. (e) RF + PUDI features. (f) RF + NeDBIT features. (g) SVM + TFO features. (h) SVM + PUDI features. (i) SVM + NeDBIT features.

Regarding the pseudo-classes, the identification performances were also satisfying using TFO and PUDI features, even if with a drop in accuracy compared with NeDBIT. This highlights the effectiveness of the APU label assignment. RF and MLP delivered the best performances. Regarding SVM, LN samples were sometimes misclassified as either WN or RN.

Overall, for P and RN classes, the NIAPU classification is almost perfect since NeDBIT features allow those classes to be properly separated from the others since they grasp the topological aspects of the set of seed genes as a whole, assigning lower and lower weights to genes that are progressively ‘far’ from the set of seed genes. For the rest of the classes, the performances are good but some genes are misclassified. This is due to the label assignment via quantiles, which obviously introduces some arbitrary noise at the boundary of such quantiles.

Results related to the other diseases are provided in Supplementary File S2, along with the results of a 5-fold cross-validation study carried out for the three sets of features.

3.2 NIAPU performances in disease gene identification

We tested the ability of NIAPU to identify new candidate genes. We performed a validation by excluding the 20% of seed genes, setting them as unlabeled both in the computation of the NeDBIT features and in the APU labeling algorithm. We repeated the procedure five times with non-overlapping gene sets. We investigated whether NIAPU was able to properly classify the removed positive genes as LP. For brevity, the results for malignant neoplasm of breast only are reported in Table 1 (other diseases in Supplementary File S2). On average, around 46% of unlabeled seed genes fell in the LP class, while the rest fell in a decreasing classification trend toward the RN class. We also observed a clear correspondence between the labeling and the association score: the higher the score, the more likely the gene is to be found in the LP class. This underlines the influence of scores on the NeDBIT features. Analogous results can be found in Supplementary File S2 for the remaining diseases.

Table 1.

Labeling of the unlabeled seed genes by NIAPU for malignant neoplasm of breast (C0006142)

Label	% Genes	Number of genes	GDAS mean	GDAS median	GDAS mode
LP	45.659 ± 1.362	93.6 ± 2.793	0.383 ± 0.016	0.346 ± 0.019	0.32 ± 0.045
WN	27.415 ± 0.636	56.2 ± 1.304	0.343 ± 0.013	0.318 ± 0.011	0.3 ± 0.0
LN	17.659 ± 4.436	36.2 ± 9.094	0.324 ± 0.012	0.303 ± 0.004	0.3 ± 0.0
RN	9.268 ± 3.65	19.0 ± 7.483	0.322 ± 0.013	0.303 ± 0.004	0.3 ± 0.0

Note: Results are intended as average with standard deviation over the five runs (GDAS: association score S).

Aggregated results related to ML classification for all the diseases are reported in Table 2. All the classes were identified by RF and MLP with high scores, while SVM reported lower metrics, particularly with regard to the LN class. Therefore, NIAPU turned out to be robust also in more challenging settings with reduced seed gene sets.

Table 2.

Classification scores as pooled mean and standard deviation (over all the diseases)

Label	Precision	Recall	F1 score
MLP
P	0.994 ± 0.011	0.998 ± 0.007	0.996 ± 0.007
LP	0.972 ± 0.013	0.972 ± 0.016	0.972 ± 0.012
WN	0.955 ± 0.02	0.915 ± 0.022	0.933 ± 0.019
LN	0.835 ± 0.021	0.744 ± 0.042	0.782 ± 0.019
RN	0.731 ± 0.037	0.86 ± 0.036	0.788 ± 0.024
Macro avg	0.898 ± 0.008	0.898 ± 0.007	0.894 ± 0.008
Weighted avg	0.884 ± 0.009	0.876 ± 0.009	0.876 ± 0.009
Accuracy	0.876 ± 0.009
RF
P	1.0 ± 0.0	1.0 ± 0.0	1.0 ± 0.0
LP	0.984 ± 0.005	0.984 ± 0.005	0.984 ± 0.005
WN	0.977 ± 0.007	0.976 ± 0.007	0.977 ± 0.006
LN	0.982 ± 0.005	0.986 ± 0.004	0.984 ± 0.004
RN	0.991 ± 0.003	0.987 ± 0.004	0.989 ± 0.003
Macro avg	0.987 ± 0.003	0.987 ± 0.003	0.987 ± 0.003
Weighted avg	0.984 ± 0.004	0.984 ± 0.004	0.984 ± 0.004
Accuracy	0.984 ± 0.004
SVM
P	0.998 ± 0.004	1.0 ± 0.0	0.999 ± 0.002
LP	0.845 ± 0.043	0.719 ± 0.071	0.767 ± 0.032
WN	0.635 ± 0.135	0.726 ± 0.108	0.625 ± 0.102
LN	0.625 ± 0.191	0.559 ± 0.026	0.419 ± 0.025
RN	0.366 ± 0.224	0.5 ± 0.004	0.38 ± 0.011
Macro avg	0.694 ± 0.066	0.701 ± 0.013	0.638 ± 0.022
Weighted avg	0.641 ± 0.077	0.642 ± 0.017	0.568 ± 0.029
Accuracy	0.642 ± 0.017

Note: Five runs were performed for each disease, masking out 20% of seed genes.

3.3 NIAPU versus other disease gene identification tools

We compared the predictive performance in the identification of candidate disease genes of NIAPU against known gene discovery algorithms, namely DIAMOnD (Ghiassian et al., 2015), Markov clustering (MCL) (Enright et al., 2002; Sun et al., 2011), random walk with restart (RWR) (Köhler et al., 2008; Valdeolivas et al., 2019), two variants of GUILD (Guney and Oliva, 2012), one exploiting the NetCombo measure and the other based on Functional Flow (fFlow) (Nabieva et al., 2005), and ToppGene (Chen et al., 2009a) (relying on the implementation provided by the GUILD software). See Supplementary File S2 for a detailed description of these algorithms. For this analysis, we relied on the extended GDA dataset provided by DisGeNET. We assigned the labels using NIAPU on the curated version of the dataset and then investigated whether the seed genes contained in the extended version (but not in the curated one) fell into the LP class. We considered the ranking retrieved by NIAPU at different quantile thresholds.

In Figure 3, we report the results of this comparison in terms of F1 score. Most of the time, our methodology outperformed or was at par with the state-of-the-art algorithms for disease gene identification, being often the best-performing method when looking for a large number of candidate genes and of comparable performances for lower ones. Indeed, DIAMOnD performs at its best when considering a low ratio (10–20%) of predicted genes, while NIAPU shows good performances both for low and high percentages of candidate genes, outperforming DIAMOnD in the latter case. In fact, as stated by the authors themselves, DIAMOnD becomes unreliable when exceeding 200 predicted genes (Ghiassian et al., 2015).

Fig. 3.

Gene discovery performances in terms of F1 score. Results are reported for six diseases for increasing numbers of candidate genes considered as a percentage of the total number of associated genes in the extended dataset, which is different for each disease. The rest of the diseases can be found in Supplementary File S2. (a) Malignant neoplasm of breast (C0006142). (b) Schizophrenia (C0036341). (c) Colorectal carcinoma (C0009402). (d) Malignant neoplasm of prostate (C0376358). (e) Bipolar disorder (C0005586). (f) Depressive disorder (C0011581).

3.4 Enrichment analysis

For a further evaluation of our results, for each of the 10 diseases considered, we performed a gene ontology/pathway/disease enrichment analysis of the first 100 predicted genes in the LP class from the validation on the extended GDA dataset. This analysis was performed using Enrichr (Chen et al., 2013; Kuleshov et al., 2016; Xie et al., 2021).

The selected LP genes do not correspond to any of the curated GDA disease genes; therefore, among the enriched diseases, we cannot expect to find the same disease for which the gene discovery process is carried out. Instead, among the enriched terms (diseases, GO terms or pathways), we should be able to find diseases and biological processes that are somehow related to the disease under scrutiny.

We report the enrichment analysis results in Table 3. In particular, we present the top enriched diseases or biological processes for each analyzed disease, together with references to literature that endorse such relevant links.

Table 3.

Enrichment analysis of the LP genes predicted for the 10 diseases of interest

Disease	Enriched disease/GO	Relationship	Reference
C0036341 Schizophrenia	KEGG GO:0042981 Regulation of apoptotic processes	Apoptotic engulfment pathway involved in schizophrenia (increased risk)	Chen et al., 2009b
C0005586 Bipolar disorder (BD)	KEGG GO:0042981 Regulation of apoptotic processes	Observed relationship between mitochondrial dynamics and dysfunction and the apoptotic pathway activation and the pathophysiology of BD	Scaini et al., 2017
C0006142 Malignant neoplasm of breast	Leukemia	Therapy-related myeloid neoplasms may be part of a cancer-risk syndrome involving breast cancer	Valentini et al., 2011
C0009402 Colorectal carcinoma (CRC)	Ovarian cancer (OC)	GCNT3 might constitute a prognostic factor also in OC and emerges as an essential glycosylation-related molecule in CRC and OC progression	Fernández et al., 2018
C0011581 Depressive disorder	Parkinson	Neurobiological investigations suggest that depression in Parkinson’s disease may be mediated by dysfunction in mesocortical/prefrontal reward, motivational and stress–response systems	Cummings, 1992
	GO:0043066 Negative regulation of apoptotic processes	Evidence of local inflammatory, apoptotic and oxidative stress in major depressive disorder	Shelton et al., 2011
C0023893 Liver cirrhosis	Parkinson	Parkinson’s disease among the neurological complications in advanced liver cirrhosis mediated by manganese	Mehkari et al., 2020
C0376358 Prostate cancer	Melanoma	Diagnoses of cutaneous melanoma may be associated with prostate cancer incidence	Cole-Clark et al., 2018
C3714756 Intellectual disability	Dementia	People with intellectual disability are at higher risk of dementia than the general population	Zigman and Lott, 2007
C0860207 Chronic alcoholic intoxication	Ovarian cancer (OC)	Alcohol consumption might be associated with the risk of OC in specific populations or in studies with specific characteristics	Yan-Hong et al., 2015
	KEGG Estrogen signaling pathway	Association of increased estrogen level and increased alcohol use in females	Erol et al., 2019
C0001973 Drug-induced liver disease	Leigh syndrome (LS)	Valproate, listed as a cause of drug-induced acute liver failure, can cause mitochondrial dysfunction and should be avoided in LS patients	Lee and Chiang, 2021

Note: The top enriched diseases and GO terms are reported, along with notes about disease relationships and main reference articles.

Although not conclusive, the fact that there is evidence in literature of links and shared biological mechanisms between the analyzed diseases and enriched diseases is additional proof of the validity and efficacy of the disease gene discovery process.

4 Discussions and conclusions

In this article, we presented the NIAPU algorithm, which fits the typical problem of the computational identification of previously unknown disease genes in the context of PU learning. The advantage of the proposed method is that it allows accurate characterization of the positive samples (P set)—via the NeDBIT features—and refined control of the LP samples (LP set)—via the APU labeling procedure—which, extracted from the set of unlabeled elements, contains, with the highest probability, elements related to the disease of interest. Moreover, NIAPU turned out to be an effective labeling procedure, allowing ML models to be trained appropriately and deliver highly accurate classification performances. As for disease gene identification, NIAPU proved to be efficient in two different experiments. In the first one, masking out a subset of seed genes, it turned out that ∼46% of those fell in the LP class. In the second one, assigning labels using NIAPU on the curated version of the DisGeNET dataset and then searching for the seed genes of the extended version only, the predictive performance of the NIAPU algorithm outperformed or was at par with the state-of-the-art algorithms for disease gene discovery.

It is worth noting that the NeDBIT features are designed to be able to use link-weighted and node-weighted graphs and that, by having increasingly accurate PPIs, we expect increasingly good results from the application of NIAPU. On the other hand, NIAPU methodology is clearly influenced by the reliability of seed genes, the association score assigned to them and the background network topology (here, the PPI network and its reliability).

Indeed, GDA datasets may be affected by disease–gene association bias due to the quantity of research on a given disease/trait. In this regard, a recent systematic review (De Magalhães, 2022) demonstrated that 87.7% of all genes could be associated with cancer. This indicates that given the massive amount of research focused on cancer, which also applies to other types of diseases, the definition ‘associated with’ is to be checked carefully and critically.

The usage of datasets that are as error-free, unbiased and reliable as possible (e.g. using an interactome validated in the specific pathological context, possibly with weighted PPIs) could potentially improve the classification performance of the method. In this regard, it is worth mentioning that an algorithm with the same theoretical ground of NIAPU has been applied in different contexts (e.g. nephrology, gastroenterology and rare diseases) (Shahini et al., 2022a, b), paying particular attention to the selection of seed genes and reference interactomes.

Funding

This work was partially supported by the ERC Advanced Grant 788893 AMDROMA ‘Algorithmic and Mechanism Design Research in Online Markets’, the EC H2020RIA project ‘SoBigData++’ [871042], the MIUR PRIN project ALGADIMAR ‘Algorithms, Games, and Digital Markets’ and by the CNR project DIT.AD021.161.001/Analisi probabilistica di dataset biologici e network dynamics.

Conflict of Interest: none declared.