INTREPPPID—an orthologue-informed quintuplet network for cross-species prediction of protein–protein interaction

Abstract

An overwhelming majority of protein–protein interaction (PPI) studies are conducted in a select few model organisms largely due to constraints in time and cost of the associated ‘wet lab’ experiments. In silico PPI inference methods are ideal tools to overcome these limitations, but often struggle with cross-species predictions. We present INTREPPPID, a method that incorporates orthology data using a new ‘quintuplet’ neural network, which is constructed with five parallel encoders with shared parameters. INTREPPPID incorporates both a PPI classification task and an orthologous locality task. The latter learns embeddings of orthologues that have small Euclidean distances between them and large distances between embeddings of all other proteins. INTREPPPID outperforms all other leading PPI inference methods tested on both the intraspecies and cross-species tasks using strict evaluation datasets. We show that INTREPPPID’s orthologous locality loss increases performance because of the biological relevance of the orthologue data and not due to some other specious aspect of the architecture. Finally, we introduce PPI.bio and PPI Origami, a web server interface for INTREPPPID and a software tool for creating strict evaluation datasets, respectively. Together, these two initiatives aim to make both the use and development of PPI inference tools more accessible to the community.

Graphical Abstract

Graphical Abstract.

Introduction

Studying how proteins work in concert has yielded deeply valuable biological insights. Not least of these insights is the implication of protein interactions in the appearance of biological phenotypes and relatedly, how disrupting these interactions can lead to disease [1, 2]. Knowledge of these interactions can, in turn, be used to design drugs and therapies to treat the diseases for which they are implicated [3, 4].

As the relevant technologies have improved, studies which experimentally verify protein–protein interactions (PPIs) have increased in size. In fact, multiple recent studies have published so-called ‘all-by-all’ studies whereby the interaction of entire proteomes are experimentally evaluated. In September 2014, researchers assembled a reference network of ~14 000 interactions from a search space of 13 000 human genes through a combination of yeast-two-hybrid (Y2H) assays and other validation experiments [5]. Five years later, the same network (‘HuRI’) expanded to ~53 000 interactions from a search space of 17 500 proteins [6].

While these studies provide an invaluable wealth of data, they represent a tremendous investment of time, labour, and money. A major consequence of the high cost of ‘all-by-all’ studies is that it is prohibitively expensive to exhaustively perform them for all known species. In fact, it is currently only practical to limit these studies to a small number of well-studied model organisms. This is a problem common to many aspects of the life sciences, which we’ve taken to calling ‘the species gap’.

As of February, 2024 there are currently 793 217 taxa spanning 108 340 genera enumerated by the NCBI Taxon Database [7]. Few, if any, experimental studies are capable of accounting for most, let alone all, of these taxa. For this reason, scientists rely on select few ‘model’ organisms that exhibit desirable properties that lend themselves to a field of study or methodology. These selection criteria further bias our already small sampling of all possible living subjects of study. To demonstrate the extent of the species gap, we quantified the number of distinct organisms that are absent from the STRING Core PPI database [8], by calculating the intersection of taxa in this database and taxa in the NCBI Taxon Database. We found that only 30% of all taxa at the class rank and 16% at the order rank were present in the STRING Core database (Fig. 1), underscoring the lack PPI information from the vast majority of species.

Figure 1.

The species gap as illustrated by the STRING PPI database. (A) the hierarchy of biological taxa [9]. (B, C) Doughnut charts indicating the proportion of all taxa represented in STRING Core at the class rank (B) and order rank (C).

Recent advances in cofractioning and cross-linking mass-spectrometry (CF-MS and XL-MS, respectively) methods have greatly increased the throughput of PPI experiments. Both methods have many advantageous properties for PPI prediction, including their speed, not requiring over-expression, and the ability to identify protein complexes [10].

In silico models of protein interaction have many favourable characteristics that may help narrow the species gap. Relative to ‘wet lab’ experiments such as Y2H or CF/XL-MS, which may take on the order of days or months to complete and analyse, in silico methods can take as little as seconds or minutes. Furthermore, most in silico PPI prediction methods operate entirely unattended, resulting in much lower labour costs. Finally, inferring PPIs is often computationally inexpensive and can leverage commodity hardware. In summary, in silico methods and their low cost lend themselves to the task of exhaustively searching protein-interaction spaces.

Many in silico methods, however, face challenges that make them unsuitable for less-studied organisms. Namely, many methods, especially those dependent on machine learning, require a training dataset comprised of PPIs. These datasets must be sufficiently large to provide a representative sample of interactions. They must also scale with the number of parameters being learned lest the model overfit. Other methods that do not rely on machine learning have similar requirements.

Regrettably, a major barrier to bridging the species gap is that there are often too few interaction data to train PPI inference models for lesser-studied organisms. In order for computational models to address the species gap, they must make accurate PPI inferences on less-studied organisms, while using existing PPI data from well-studied organisms during training. Such ‘cross-species’ inference can be used to infer the intraspecies PPI network of the less-studied organism, all while only using the model organism as training data, addressing the species gap. Cross-species inference requires that methods perform accurate out-of-distribution (OOD) inference. Historically, PPI inference methods have failed to make accurate predictions on interactions between OOD proteins [11–13].

In 2012, a letter published by Park and Marcotte outlined the severity of pervasive data leakage that was obscuring the inability of methods of the time to generalize to interactions between OOD proteins [11]. The cause of the data leakage Park and Marcotte observed was the common ‘random splitting’ approach to preparing cross-validation folds. Random splitting of interactions (between pairs of proteins) into folds allows for the same proteins appearing in both the training and testing sets, resulting in data leakage. Park and Marcotte found that when methods of the time were trained and tested on ‘strict’ datasets where proteins present in the training set never appear in any of the other folds, the accuracy of most methods dropped to random levels. This precipitous drop in performance indicates that all the models tested had problems dealing with OOD proteins.

While Park and Marcotte found that OOD performance was poor across the landscape of methods available in 2012, studies by Dunham and Ganapathiraju in 2021 and Bernett et al. in 2023 corroborate the 12-year-old findings [12, 13]. In both studies, contemporary sequence-based PPI inference methods were tested on datasets that satisfy Park and Marcotte’s strictest validation schemes. Predictive power dramatically degraded in all cases, just as Park and Marcotte observed in 2012. While models for PPI inference have changed drastically since 2012 with the notable introduction of deep learning methods, reported performance of methods continue to be inflated as a result of the random-splitting of interactions (as opposed to stricter splitting based on proteins).

Furthermore, work by Hamp and Rost as well as Bernett et al. underscore an additional source of data leakage: distinct proteins with sequences which are nearly identical [13, 14]. Amino acid sequences can be overwhelmingly redundant between distinct proteins. An important consequence of this is that two proteins with all but identical sequences may appear in both sets, thereby leaking data in the same way randomly-splitting PPI pairs does. The challenges presented by sequence similarity can be overcome by ensuring that proteins in the training set do not have a sequence identity above some threshold with proteins in the testing set.

One of the earliest approaches for sequence-based PPI inference is Marcotte et al.’s ‘Rosetta Stone’ method [15]. The authors outline a theory whereby interacting proteins of one species sometimes appear as fused homologous proteins in other species. Using this technique, Marcotte et al. inferred interactions between proteins from Escherichia coli using information of proteins of other genomes that had similar sequences to the putative pair. The concept of ‘interologue’ was soon after introduced to describe proteins whose interactions are conserved among their respective orthologues [16]. Interologues allow us to extend knowledge of interactions of one species to other species by identifying both sequences and interactions that are conserved between them [17, 18]. These homology-based methods, while effective at identifying conserved interactions, have limitations related to unknown interactions, as acknowledged in the work that first introduced the term interologue: ‘Although potentially powerful, applying the interologue concept alone in estimating the significance of potential interactions precludes the finding of novel connections previously unidentified in other model organisms’ [16].

Some of the first machine-learning methods for predicting PPIs were kernel-based methods operating on sequence features. The support vector machine (SVM) is an algorithm for finding the maximum-margin hyperplane for a classification or regression problem, relying on kernel transformations of the input data to solve nonlinear problems [19]. Effectively applying SVMs to the problem of PPI classification depends greatly on the design of the kernels used to represent protein pairs. These kernels, designed by expert knowledge, may compare individual proteins to each other (‘genomic kernels’) or pairs of proteins to other pairs (‘pairwise kernels’). Furthermore, these kernels may compare a variety of features of the gene products being studied, including motifs, gene ontology (GO) annotations, sequence similarity, or some combination thereof [20].

As advances in hardware and algorithms made research into deep learning more feasible, they’ve proven to be effective in a number of problem domains, not least of which being PPI prediction. Richoux et al. demonstrated that twin neural-networks are far more efficient at PPI inference than other network architectures [21]. Twin neural networks are models that consider pairs of inputs using the same trainable parameters rather than specifying parameters for each input and are commonly used to make comparisons between two inputs [22, 23]. Twin networks can reduce the risk of overfitting and aiding few-shot performance [24]. Many twin PPI prediction methods rely on convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to encode feature sequences. The PIPR method by Chen et al., however, eschews these architectures for a hybrid: the recurrent CNN (RCNN) [25]. The RCNN comprises of an initial CNN layer whose output is fed into a bidirectional gated recurrent unit (GRU) RNN [26].

We’ve previously described RAPPPID, a method for PPI prediction that outperforms leading methods on strict validation sets that conform to the guidelines set out by Park and Marcotte [11, 27]. RAPPPID is a twin network that learns latent representations of sequences and inputs them into a classifier network to infer whether two proteins interact. RAPPPID achieves its high performance largely due to its adoption of averaged weight-decayed LSTM (AWD-LSTM) and the Unigram tokenization algorithm [28, 29]. The former is a combination of regularization techniques that are effective for long short-term memory (LSTM) RNNs. The input is tokenized using subword regularization, which regularizes the network by sampling from a distribution of different segmentations of the input [30].

In spite of the aforementioned progress in the field, most computational models are not designed to address the species gap and exhibit reduced performance when tested on the cross-species task. This makes them unsuitable to address the species gap problem and as such, the knowledge of the protein interactions that govern fundamental biological processes of ill-studied nonmodel organisms remains beyond our grasp. To address this issue, we introduce an extension of RAPPPID named INTREPPPID (Incorporating Triplet Error for Predicting PPIs using Deep Learning). INTREPPPID is specifically designed to improve cross-species PPI prediction performance with the aim of reducing the species gap. In this study, ‘cross-species’ PPI prediction refers to predicting the intraspecies interactions of proteins of one species using a computational model that is trained on intraspecies interactions of a different species. This term does not refer to predicting the interaction between one protein from one species and another protein from a different species.

We designed INTREPPPID to learn latent representations of amino acid sequences that are useful for predicting PPIs as well as identifying orthologues. Alongside RAPPPID’s original twin network for inference, INTREPPPID adds a triplet network that minimizes the Euclidean distance between latent representations of orthologous sequences while maximizing it for nonorthologous sequences. Parameters are shared between both the twin and triplet networks, resulting in five networks with a single set of shared parameters, which we call a ‘quintuplet’ network. INTREPPPID’s architecture learns a latent space that encodes both PPI and orthology information. This allows us to enforce an inductive bias loosely based on the concept of interologues.

The contributions of this study are as follows. First, we introduce INTREPPPID, a deep learning method based on a novel quintuplet architecture and a carefully designed loss function that makes out-of-distribution predictions of PPIs to address the species gap. Our analysis shows that INTREPPPID outperforms all leading methods tested on both intraspecies and cross-species PPI predictions. Second, we demonstrate that INTREPPPID effectively learns the orthology of proteins and its performance improvement is due to learning biologically relevant information. This suggests the use of orthology information as a powerful approach for making out-of-distribution predictions in this domain. Third, we have released to the community PPI.bio, an open-source interface for INTREPPPID and RAPPPID hosted on the internet. This allows researchers to infer PPIs without installing any specialized software or using the command line. Finally, we have released PPI Origami, a computer programme for creating reproducible strict PPI databases in an automated fashion. By automating the dataset generation process, we hope to reduce common sources of data leakage in datasets used for PPI inference.

Materials and methods

The INTREPPPID architecture

INTREPPPID integrates RAPPPID’s twin network for the PPI Classification task with a triplet neural network used for a novel Orthologous Locality task. Learning the two networks jointly results in five encoder layers with shared parameters in what we’ve come to call a quintuplet network architecture.

INTREPPPID uses the same AWD-LSTM encoder as the one used in RAPPPID to learn vector representations of protein sequences. This encoder is a regularized RNN that has been shown to improve performance on tasks that require inferring interactions between proteins absent from the training set [27]. Before encoding, sequences are first tokenized using the Unigram algorithm as implemented by the SentencePiece library [29, 30]. INTREPPPID uses the vector representations generated by the encoder for two downstream tasks: the ‘orthologous locality’ task and the ‘PPI classification’ task. The encoder as well as the two tasks are trained in an end-to-end fashion.

INTREPPPID’s quintuplet architecture means that training the network relies on additional orthology data. At inference time, however, only the protein classification twin network is used, so no information other than the amino acid sequences of the proteins in question are required as input.

The protein–protein interaction classification task

INTREPPPID infers PPIs by learning latent representations of the candidate protein sequences which can be classified by a fully connected neural network outputting the interaction probability. The two amino acid sequences ( and ) are first tokenized into vectors and using the Unigram algorithm. The Unigram algorithm segments the amino acid of the protein sequence into groups using a vocabulary of 250 tokens. The Unigram algorithm also employs subword regularization. Unigram draws tokens for a given sequence from a probabilistic distribution of all possible combination of tokens in the vocabulary, meaning that the same sequence can be tokenized differently between successive calls to the tokenizer.

Once the tokenized vectors and are obtained, they are inputted into an encoder () that outputs a latent vector for each protein ( and , respectively):

The encoder has an AWD-LSTM architecture, that leverages drop-connect and averaged optimization (here, stochastic weight averaging), among other things, to regularize and improve generalization (see RAPPPID’s study for details [27]). The parameters of the encoder that infers from are the same as those of the encoder that infers from , making the PPI classification network a twin neural network.

Finally, the latent vectors and are averaged and inputted into a classifier network that outputs an interaction probability :

The classifier is a two-layer, fully connected neural network whose hidden layer is activated by the Mish function [31]. The fully connected neural network outputs a single real number which, once activated by the sigmoid function, represents the probability that the proteins represented by the amino acid sequences and interact. This interaction probability as well as the true label are inputted into the binary cross-entropy loss to obtain a loss for the PPI classification network:

where is a binary label representing whether the two proteins truly interact () or not ().

The orthologous locality task

INTREPPPID introduces a novel task to help inform the PPI classification network. The orthologous locality task is to learn a latent space whereby the Euclidean distance between latent representations of proteins that are orthologous to one another are within some margin . Conversely, nonorthologous proteins are to have a Euclidean distance greater than .

To achieve this, we adopt a triplet network and a triplet margin loss following the description of [32]:

where is the latent vector of some ‘anchor’ protein for which there is a known orthologous protein with a latent vector , is the latent vector of a randomly chosen protein that is not an orthologue of , is a margin factor and is the Euclidean distance:

For the experiments conducted herein, the margin was fixed at a value of 1.

The principle of the triplet network is to, at every training step, maximize the distance between two nonorthologous proteins (the ‘anchor’ and ‘negative’ proteins), as well as minimize the distance between two orthologous proteins (the ‘anchor’ and ‘positive’ proteins).

The anchor protein, with amino acid sequence , can be any protein in the dataset for which orthology data are known. All orthology data for the experiments conducted herein are from the Orthologous Matrix (OMA) database of orthologues [33]. In the event that orthologues are known for the protein that encodes, is chosen to be .

The positive and negative proteins (with amino acid sequences and , repectively) are chosen relative to . is chosen to be the sequence of a randomly chosen protein that, according to the OMA database, belongs to the set of sequences of proteins orthologous to the anchor protein. At every training step, is drawn randomly from this set of proteins orthologous to , meaning that the positive protein for a given anchor protein may not be the same between training steps or epochs. The negative protein is sampled in a similar manner to the positive protein, but rather than being sampled from the set of proteins orthologous to , is sampled from the set of proteins not orthologous to . The amino acid sequence of the anchor, positive, and negative proteins (, , and ) are tokenized and encoded in the same manner as and , resulting in their latent representations , , and .

Optimizing the triplet margin loss in this manner maximizes the Euclidean distance between latent vectors of proteins that are not orthologues, while minimizing the distance between vectors of orthologues. Our adoption of the triplet margin loss follows evidence that such a loss is an effective way of creating feature descriptors that are ‘spread out’ over the latent space, which in turn increases performance on downstream classification tasks [32, 34]. Furthermore, orthologues typically share the same protein functions and therefore offer a good interspecies basis for the downstream PPI classification task [35, 36].

INTREPPPID’s loss function

The final INTREPPPID loss function that is optimized at training time is the sum of and ,

In this way, INTREPPPID optimizes both the orthologous locality and PPI classification losses simultaneously, in an end-to-end fashion, ensuring that latent vectors simultaneously encode orthology and PPI information.

Preparing strict protein–protein interaction datasets

Experiments were run using datasets with strict data splitting folds, described as ‘C3’ by Park and Marcotte [11]. Namely, we ensured that there was no overlap of proteins between any of the training, validation, or testing datasets. Additionally, we controlled for sequence similarity of proteins between the training, validation, and testing sets that is a known vector for information leakage [13, 14]. We specifically control to ensure that proteins with >90% sequence identity are not present between splits to ensure that proteins that differ only by name are not accidentally present in two different splits. Another reason for choosing a threshold of 90% sequence identity is that it allows us to test the ability of INTREPPPID to learn motifs and domains relevant to interactions. Individuals who wish to create testing datasets that limit the exposure of similar motifs and domains can create those with more strict sequence identity requirement (e.g. a 50% threshold) using PPI Origami (see Generating Strict Protein–Protein Interaction Datasets with Protein–Protein Interaction Origami). Protein interaction data are obtained from the STRING database, version 12 [8]. Interactions with a confidence score <90% are filtered. Supplementary Methods provide details of how we created such strict datasets, and Supplementary Table S1 outlines summary statistics for the datasets used in this study.

Generating strict protein–protein interaction datasets with protein–protein interaction origami

We have written a command-line interface (CLI) tool named ‘PPI Origami’ with the aim of making the task of preparing strict PPI datasets in a replicable and automated fashion. PPI Origami has three primary functions: ‘downloading’ primary databases, ‘processing’ primary databases into strict datasets, and an ‘analysis’ function.

PPI Origami allows for the downloading of primary databases as an integrated part of its pipeline. By directly handling downloading third-party databases, PPI Origami ensures that different users of the software are downloading the same files in the same way. PPI Origami also stores downloaded files in a distinct location from the processed files, isolating primary files from accidental changes.

PPI Origami’s main function is to process and transform primary databases into strict PPI datasets. It can create datasets that satisfy both criteria of strict PPI datasets, filtering low-confidence positive examples, and drawing negative examples as previously described.

PPI Origami allows for simple reporting of basic statistics of PPI datasets it generates. These reports allow for convenient verification of whether datasets meet both criteria of strict datasets, as well as other statistics of interest such as the ratio of interactions between dataset cross-validation subsets.

We hope that PPI Origami encourages other researchers to incorporate strict PPI datasets in the future development and analyses of supervised PPI inference methods.

Training procedure

INTREPPPID was trained using the Ranger21 optimizer, which uses AdamW with a look-ahead mechanism [37, 38]. An ‘explore-and-exploit’ learning rate scheduler was used, with a maximum learning rate of 0.01 [39]. INTREPPPID was trained for 100 epochs. The weights of the epoch with the lowest validation loss were used in the analyses herein (and for evaluation on the test set). Hyperparameters used to train INTREPPPID were informed by those selected during the development of RAPPPID and are listed in Supplementary Table S2.

Results

INTREPPPID outperforms leading models for prediction of human protein–protein interaction

First, we set out to evaluate the performance of INTREPPPID (Fig. 2A) in an intraspecies PPI prediction task: trained and tested on Homo sapiens interactions in the STRING database (see Materials and Methods). Table 1 shows the performance of INTREPPPID against five leading PPI prediction methods. These methods include RAPPPID, our previous method upon which INTREPPPID expands, D-SCRIPT, a method designed for the cross-species task, PIPR, a twin network with an RCNN encoder, SPRINT, a sequence-similarity-based method, and the twin networks proposed by Richoux et al. [21, 25, 27, 40, 41]. INTREPPPID performed slightly better than RAPPPID across all metrics and outperformed all other methods (Table 1, Fig. 2B, Supplementary Table S3). Richoux et al., D-SCRIPT, and SPRINT perform similarly on average, with D-SCRIPT exhibiting a particularly high degree of variability across the different datasets tested. We observed severe over-fitting while training and testing PIPR on all datasets. Precision–recall and receiver-operator Curves are reported in Supplementary Figs S1 and S2, respectively.

Figure 2.

INTREPPPID’s architecture and performance. (A) A schematic overview of INTREPPPID’s quintuplet architecture. The leftmost panel depicts the triplet network responsible for the orthologous locality task, while the rightmost panel depicts the triplet network responsible for the PPI classification task. During training, INTREPPPID learns both tasks jointly through an overall loss function, which is the sum of the losses of the two tasks. (B) Test (C3) Matthew’s correlation coefficient (MCC) of leading PPI inference methods trained and tested on H. sapiens datasets.

Table 1.

Average performance of five leading PPI inference methods trained and tested on four H. sapiens PPI datasets. The performance metrics are calculated on strict C3 test sets (see Materials and Methods). The upward arrow (▲) indicates higher values are better for a metric, whereas a downward arrow (▼) indicates lower value are better. Boldface indicates the best value. The following abbreviations are used: AUROC: area under the receiver operating characteristic; AP: average precision; MCC: Matthew’s correlation coefficient, SD: standard deviation.

Method	INTREPPID	RAPPPID	D-SCRIPT	Richoux et al.	SPRINT	PIPR
▲ AUROC Mean (±SD)	0.752 (± 0.017)	0.732 (± 0.024)	0.634 (± 0.125)	0.606 (± 0.009)	0.597 (± 0.061)	0.482 (± 0.006)
▲ AP Mean (±SD)	0.730 (± 0.022)	0.717 (± 0.015)	0.627 (± 0.086)	0.614 (± 0.015)	0.641 (± 0.064)	0.484 (± 0.003)
▲ MCC@50% Mean (±SD)	0.365 (± 0.067)	0.313 (± 0.041)	0.153 (± 0.144)	0.161 (± 0.017)	0.133 (± 0.096)	−0.030 (± 0.015)
▲ F1@50% Mean (±SD)	0.697 (± 0.019)	0.630 (± 0.025)	0.506 (± 0.282)	0.370 (± 0.030)	0.566 (± 0.048)	0.318 (± 0.010)
▼ Brier Score Mean (±SD)	0.200 (± 0.004)	0.215 (± 0.006)	0.402 (± 0.116)	0.274 (± 0.011)	0.493 (± 0.003)	0.487 (± 0.005)

Unlike our benchmarks, it is estimated that there are far more pairs of proteins that do not interact than those that do [6, 42]. Accordingly, we tested INTREPPPID’s performance at increasing negative-to-positive interaction ratios and found that no appreciable change in performance was found (Supplementary Table S4).

INTREPPPID maintains high cross-species performance on multiple organisms

INTREPPPID was designed to improve on RAPPPID’s ability to make predictions across species. We therefore sought to compare INTREPPPID’s cross-species performance with that of RAPPPID’s as well as other leading PPI prediction methods.

When trained on H. sapiens and tested on various species (C3), ranging from Mus musculus (house mouse) to Arabidopsis thaliana (thale cress), INTREPPPID predominantly outperformed other methods (Fig. 3A, Supplementary Table S5), with only RAPPPID rarely surpassing its performance in some metrics. Figure 3A shows the mean AUROC (across three datasets) of different methods, with INTREPPPID significantly outperforming all other methods (one-sided Wilcoxon signed rank test versus RAPPPID = 0.03). On the cross-species task, the performance of PIPR and SPRINT were close to a random classifier. Both D-SCRIPT and the Richoux et al. method performed similarly to one another, but consistently scored below RAPPPID and INTREPPPID.

Figure 3.

(A) Average cross-species AUROC of leading PPI inference methods. The y-axis shows the AUROC of each method (shown as different colours) against the AUROC of RAPPPID (x-axis). Each model was trained on H. sapiens and tested on various species (indicated by the different markers). Markers above the dashed grey line indicate instances where a method outperformed RAPPPID, while markers below the line indicate under-performance. (B) The precision of inferred interactions among the k^"th" percentile of the scores inferred by the method indicated by the line colour. The x-axis shows k and the y-axis shows precision along the y-axis. Interactions were inferred for datasets from seven different species. The mean across these datasets is reported by the solid line.

In many applications, one is particularly interested in the precision of a model among the highest-ranking predicted edges (e.g. to validate them experimentally). To evaluate INTREPPPID’s suitability for such tasks, we measured the precision of predictions whose scores are above various percentiles (Fig. 3B). Considering the average across all species tested, INTREPPPID achieved a precision >90% among predictions above the 85th percentile, and even when we considered the 75th percentile, its mean precision remained >85%. These observations confirm that INTREPPPID is suitable for applications where very high precision is paramount.

We sought to further confirm INTREPPPID’s ability to generalize by measuring the number of interactions in additional PPI databases that INTREPPPID was capable of recovering. When considering interactions with confidence scores of 95% or higher, INTREPPPID was capable of correctly identifying 72% of interaction in the HIPPIE database and 69% of interaction in the IntAct database. These results are only slightly lower than the 74% true-positive rate INTREPPPID achieves on our benchmark dataset.

INTREPPPID learns protein representations that exhibit strong orthologous locality

Through its orthologous locality task, INTREPPPID learns latent representations of orthologues that are near to one another in the Euclidean space, while ensuring latent representations of nonorthologous sequences remain distant. To qualitatively assess the locality of orthologues in latent space, we calculated the embeddings of the sequences of 1000 randomly sampled proteins present in the OMA database of orthologues and projected them onto a 2D space using UMAP and PCA (Supplementary Fig. S3) [43]. The projections show that embeddings of orthologues cluster tightly in INTREPPPID’s latent space, while embeddings from RAPPPID (which has a similar architecture, but without the triplet network) do not.

A more quantitative analysis of embedding distances further confirmed that INTREPPPID’s latent space exhibits strong orthologous locality. Namely, the Euclidean distance between latent representations of orthologues was on average 0.822, while the distance between the representations of nonorthologous proteins is on average 5.31 (Supplementary Fig. S3C). Nearly 80% of INTREPPID’s latent representations of orthologues fell under the triplet loss margin of 1, while virtually all nonorthologues had a distance greater than the margin between them.

Together, these analyses confirm that INTREPPPID successfully accomplishes the orthologous locality task.

INTREPPPID’s performance gain is due to the biologically relevant information encoded by orthology

To further assess the role of orthology information on the performance of INTREPPPID, we conducted an experiment in which we trained INTREPPPID on two separate datasets: one constructed using orthologue data from the OMA database (called ‘not scrambled’) and one where ‘random’ pairs of proteins from the OMA database were labelled orthologues (‘scrambled’). After training and testing on both variants of the H. sapiens dataset, we found that INTREPPPID’s performance dropped to near random in the ‘scrambled’ case (Supplementary Table S6, AUROC = 0.524). Moreover, on this dataset, neither the classification loss nor the triplet loss significantly changed over 30 000 training steps (Supplementary Fig. S3D).

INTREPPPID’s low testing performance on ‘scrambled’ data suggests that the mere exposure of sequences from various species is not sufficient to increase PPI classification performance. The impact of the ‘scramble’ dataset on both the classification and triplet losses suggests that the orthologous locality task is complimentary to the PPI classification task. Ablating the biological meaning of the orthology data conversely results in the orthologous locality task negatively impacting PPI classification performance. These results show that the encoding of orthology information increases INTREPPPID’s performance due to its biological relevance.

Having established that the central contribution of incorporating orthologues in the INTREPPPID model is the biological information they encode, we continued to test the extent to which they are relevant to INTREPPPID’s performance. To do so, we constructed INTREPPPID datasets of H. sapiens protein interactions much as we have for previous cross-species analyses herein. For this analysis, however, orthology data related to proteins belonging to any of the testing species (H. sapiens, M. musculus, Danio rerio, Drosophila melanogaster, Caenorhabditis elegans, Saccharomyces cerevisiae, A. thaliana) were omitted. Relative to INTREPPPID trained on dataset that included all available orthology data, the altered dataset resulted in worse testing performance metrics across every species tested (Supplementary Table S7). The decrease in performance was least severe when testing on H. sapiens (i.e. intraspecies inference), having measured a −9.00% change in MCC and −1.36% change in AP scores. Conversely, the performance drop was much greater on testing sets of species different from the training species (i.e. cross-species inference), with observed changes as low as −38.85% for MCC and −17.58% for AP scores in some instances. This further confirms that orthology information is used by the model to make predictions.

INTREPPPID learns interactions of evolutionarily conserved functions that RAPPPID does not

To better understand the biological characteristics of the interactions that account for INTREPPPID’s performance gains and the role that its orthology loss plays, we analysed the pairs of interacting proteins from our testing datasets in which INTREPPPID predicted a higher probability of interaction than RAPPPID by the widest margin. We found that, on average, INTREPPPID assigns higher scores to interacting pairs in 73% (±11%) of all cases. Of those inferred interactions, the top interactions were disproportionately between proteins that are evolutionarily conserved between species.

Our analysis began with pairs of proteins in the testing dataset that interact and were correctly identified by INTREPPPID (i.e. a score ≥0.5), and incorrectly identified by RAPPPID (i.e. a score <0.5). To examine cases in which most typified INTREPPPID are performing well while RAPPPID are falling short, we further filtered the aforementioned interactions such that only protein pairs with the largest differences between the INTREPPPID and RAPPPID scores remained (i.e. in the 90th percentile or higher). These scored protein pairs will be referred to as ‘top INTREPPPID-unique positive hits’ hereafter.

A statistical analysis of over-representation of Gene Ontology (GO) Molecular Function annotations was performed on the top INTREPPPID-unique positive hits using the PANTHER Classification System [44–46]. This analysis provided us with an understanding of which GO Molecular Function annotations were over-represented among proteins that comprised the top INTREPPPID-unique positive hits, relative to those in the testing set as a whole.

We found that, across the species tested, the GO Molecular Function annotations that were overexpressed were primarily regarding ubiquinone activity, ribosome structure, and RNA binding (Supplementary Table S8). These functions all belong to pathways that are consistently found to be highly conserved across a broad spectrum of species [18, 47, 48].

An example of one of these interactions is one between the large ribosomal subunit proteins uL24A and uL10 found in S. cerevisiae encoded by the genes RPL26A and RPP0, respectively. INTREPPPID correctly predicts that these two proteins interact with a probability of 94.6% while RAPPPID assigns a probability of 4.52%, both models having been trained exclusively on interactions between human proteins. The two proteins in question are highly conserved, with 131 and 798 orthologues recorded in the OMA database for each protein, respectively. In both cases, a human orthologue is identified by OMA (RPL26L1 and RPLP0 genes, respectively).

These observations suggests that INTREPPPID correctly identifies interactions that have functions crucial to pathways such as protein degradation and protein translation that are highly conserved between species, that RAPPPID does not. In addition, these observations provide further support that the discrepancy between INTREPPPID and RAPPPID’s performance among interactions composed of highly conserved proteins is driven by the similarity of embeddings among orthologous proteins (i.e. the orthologous locality property), which is the primary distinction between INTREPPPID and RAPPPID.

A web server and interface for RAPPPID & INTREPPPID

While we have made best efforts to make RAPPPID and INTREPPPID as simple as possible to run on a local computer with minimal computational requirements, it currently requires knowledge of the CLI and the Python computer language. These prerequisites, while accessible to researchers with relevant experience, can be too great a hurdle for others.

To address this, we have created a website which acts as a graphical interface for a server that is capable of performing proteome-scale inference in under 15 min (https://ppi.bio). The source code that runs PPI.bio is available publicly on GitHub and can be found by visiting the internet address: https://github.com/Emad-COMBINE-lab/ppi.bio. Many online academic servers are no longer available online and do not provide their source openly, resulting in the total loss of these servers (see Supplementary Table S9 for a review of 22 such resources, 50% of which are no longer alive). We hope to enable others to run their own server in the event that https://ppi.bio goes offline by releasing its code and documentation.

The process for making proteome-wide inferences on user-submitted amino acid sequence is as follows. Users input amino acid sequences that are then submitted to the PPI.bio server (Fig. 4). The PPI.bio server in turn encodes the sequence using the user’s choice of either RAPPPID or INTREPPPID’s encoder to obtain a protein embedding. The embedding is then paired with precomputed embeddings of all proteins belonging to the proteome of the user’s choosing (currently, proteins for six organisms have been computed). The probability of these pairs of proteins interacting are then inferred using RAPPPID or INTREPPPID’s classification head (based on the user’s choice), and returned to the user’s browser window.

Figure 4.

The webserver of PPI.Bio supporting RAPPPID and INTREPPPID at whole proteome scale. (A) Overview of the operation of the PPI.Bio PPI inference server. In the first stage (leftmost panel), amino acid sequences are encoded into protein embeddings using either RAPPPID or INTREPPPID, depending on the user’s preference. In the second stage (rightmost panel), precomputed protein embeddings of a specified species proteome are loaded from a database. The classification head of either RAPPPID or INTREPPPID is then used to compute the probability of interaction of the user-specified protein and the precomputed embeddings. Screenshot of the (B) data table of interaction probabilities, (C) probability histograms, and (D) GO enrichment analysis powered by PANTHERDB, as seen on PPI.Bio after inferring proteome-wide interactions.

PPI.bio also enables users to conduct GO enrichment analysis on the set of proteins that have an interaction probability greater than some user-specified threshold. PPI.bio uses the PANTHER online API [44] to calculate GO enrichment statistics. This functionality allows users to find instances of GO annotations being statistically over-represented among proteins that were inferred to interact with the protein supplied by the user.

Discussion

This study introduces three tools for researchers in their pursuit of better understanding PPIs: a cross-species method for PPI inference (INTREPPPID), a software package for creating strict PPI datasets (PPI Origami), and a web server for inferring PPIs using INTREPPPID and RAPPPID (PPI.bio). These tools and their code are all freely available under open licenses with complete documentation.

We view INTREPPPID’s novel contributions as being two-fold. First, INTREPPPID is among the few sequence-based PPI inference methods that are specifically designed with the purpose of addressing the challenge of accurately inferring protein interactions between proteins of species other than that it was trained on (i.e. cross-species inference). The only other such methods specifically designed for this purpose known to us are D-SCRIPT and PIPE4 [40, 49]. Of these two methods, only D-SCRIPT’s software is made publicly available, which has made its inclusion in the analyses herein possible. We believe that INTREPPPID joining D-SCRIPT in being the only publicly available sequence-based cross-species PPI inference methods represents a significant contribution to the community.

Secondly, INTREPPPID introduces a novel orthologous locality loss and quintuplet architecture which, as we demonstrated, is responsible for a significant increase in both intra- and cross-species PPI inference performance under strict evaluation conditions. While INTREPPPID has a quintuplet network, there is no appreciable increase in trainable parameters compared to RAPPPID’s twin network construction due to the parameter sharing between encoders (2.20E5 and 1.88E5 parameters, respectively). By sharing parameters, INTREPPPID reduces both its GPU memory footprint as well as the risk of over-fitting via over-parameterization.

The orthologous locality task can be viewed through the lens of contrastive self-supervised learning (SSL) techniques like ‘SimCLR’ or Time-Contrastive Networks [50, 51]. Specifically, orthologous sequences can be thought of as multiple views of the same protein augmented by an evolutionary process in a way that is analogous to how SimCLR compares pairs of images that have been augmented by, e.g. cropping or blurring. In both instances, the model is learning invariance between entities that are similar in their desired feature space, but are significantly different in the input space. By creating a feature space that is invariant to the effects of speciation on amino acid sequences, we position our model to capture more interologues. While SimCLR only uses pairs of contrasting examples, SSL methods like time-contrastive networks have adopted a triplet loss to more efficiently learn embeddings that better fit downstream tasks.

Sequence-based methods rely on a large number of PPI examples to learn to recognize features in pairs of amino acid sequences that indicate two pairs of proteins are likely to interact. The more examples provided, the better models become at identifying features that we know from experimental knowledge are important for mediating interactions.

As is the case with all sequence-based methods for PPI inference, INTREPPPID risks becoming brittle and only memorizing sequences found in the training set or learning trivial solutions such as reporting the sequence similarity of two proteins as their likelihood of interaction. In the absence of more experimental data to train on, ensuring the data are of high-quality, validating strictly and judicious application of regularization are the primary ways to address these generalization concerns. Our previous work on RAPPPID underscores how these methods can be used to create generalized, performant models that do not rely on trivial solutions like comparing sequence similarity nor memorizing sequences.

While INTREPPPID is shown here to outperform leading methods in the intra- and cross-species tasks, it is important to note that deep learning, sequence-based methods like ours can sometimes struggle with the changes observed by certain protein variants and isoforms. Future work might help address this by constructing architectures that provide inductive biases relevant to learning binding domains and residues.

Additionally, all of the sequence-based methods tested infer interactions that occur under unspecified intrinsic (e.g. protein conformation, post-translational modifications) and extrinsic (e.g. subcellular localization, pH, temperature) conditions (a property inherited from the training STRING dataset). While databases such as Reactome richly annotate interactions, including information about, e.g. PTMs and subcellular localizations, the number of interactions and organisms in the database is limited due to the high-labour costs associated with manual curation [52]. Future work towards reconciling and augmenting existing PPI datasets and methods with such annotations could possibly help to provide valuable context to the interactions provided.

The second tool presented in this work is PPI Origami, a framework for creating new strict PPI datasets. While many reference or ‘gold-standard’ PPI datasets exist, they each bring with them different underlying assumptions and limitations [11–13]. Researchers might wish to take advantage of large, comprehensive PPI databases like the STRING database used to create the aforementioned strict PPI datasets or alternatively take a specialist approach and interrogate smaller curated networks that describe certain organisms, diseases, or cell types [53–55]. They may also wish to derive their training set from one dataset and their evaluation datasets from another, much as we did over the course of our experiments. PPI Origami provides tools for creating PPI datasets from sources the user specifies, while automating the process of ensuring that the data conform to the criteria outlined herein. By providing tools and frameworks rather than one set of datasets, we hope to impower researchers to adopt PPI datasets that (i) do not leak data and (ii) conform to the specific needs of the researcher.

The final major contribution herein is PPI.bio, a web server for inferring PPIs using RAPPPID and INTREPPPID. PPI.bio infers interactions entirely on the Central Processing Unit (CPU), greatly reducing server costs. Both RAPPPID and INTREPPPID facilitate this by allowing known protein sequences to be precomputed, only requiring a forward pass of the shallow fully connected classification network when the embeddings for both proteins has already been calculated. This allows for significant speed-ups when doing proteome-wide inferences.

As time passes, online services published in academic articles tend to go offline or otherwise become inaccessible. After conducting a search for the terms ‘protein-protein interaction portal’ and ‘protein-protein interaction website’ on the online academic search engine SemanticScholar.org and randomly sampling 22 articles that introduce online tools, 45% of the URIs either did not resolve or no longer offered the service introduced (Supplementary Table S9). To help ensure that PPI.bio remains accessible for years to come, we’ve made the code that runs PPI.bio available under a free and open license. To help facilitate the deployment of PPI.bio, the server is designed to be easily deployed to the Platform as a Service (PaaS) service Heroku as well as the self-hosted alternative Dokku.

Key Points

• Most PPI studies are conducted in a select few organisms, leading to a ‘species gap’.

• Performant cross-species PPI methods are required to bridge this species gap.

• We introduce INTREPPPID, which improves cross-species performance using a novel quintuplet architecture to incorporate orthology data.

• INTREPPPID outperforms leading PPI inference methods on both intra and cross-species tasks across multiple metrics.

• We also have released an online interface for INTREPPPID (PPI.bio) and a tool for making PPI datasets (PPI Origami).

Author contributions

A.E. and J.S. contributed to project conceptualization, methodology as well as writing, editing, and reviewing. A.E. contributed to supervision and funding acquisition. J.S. contributed to software, data curation, investigation and visualization. All authors read and approved the final article.

Funding

This work was supported by grants from Natural Sciences and Engineering Research Council of Canada (NSERC) [RGPIN-2019-04460] (A.E.) and Canada Foundation for Innovation (CFI) JELF [project 40781]. This research was enabled in part by support provided by Calcul Québec (www.calculquebec.ca) and the Digital Research Alliance of Canada (alliancecan.ca).

Software & data availability

INTREPPPID and the PPI.bio server are freely available under the Affero General Public License version 3.0. PPI Origami is licensed under the GNU General Public License version 3.0. They can be downloaded from https://github.com/Emad-COMBINE-lab/intrepppid, https://github.com/Emad-COMBINE-lab/ppi.bio, and https://github.com/Emad-COMBINE-lab/ppi_origami, respectively.

All code and datasets have been deposited in the Zenodo repository under record numbers 10652231, 10652229, 10652233, and 10594149.