Protein language models are accidental taxonomists

Abstract

Protein-protein interactions (PPIs) are fundamental to nearly all biological processes, yet their experimental characterization remains costly and time-consuming. While computational methods, particularly those using protein language models (pLMs), offer higher-throughput solutions, they often report unexpectedly high performance on multi-species datasets. Here, we introduce the accidental taxonomist hypothesis, proposing that neural networks can exploit the phylogenetic distances across labels in protein datasets rather than genuine interaction features. We show that in standard multi-species PPI datasets, positive pairs typically share a taxonomic origin, while randomly sampled negatives do not. We then demonstrate that pLM embeddings can be used to accurately distinguish whether two proteins share a taxonomic origin, allowing models to “cheat” by learning phylogeny instead of genuine PPI features. By employing a strategic sampling strategy that restricts negative examples to protein pairs from the same species, we reveal a marked drop in model performance, confirming our hypothesis. Compellingly, these strategically trained models still outperform single-species models, suggesting that multi-species data can improve performance if carefully curated. These findings suggest that accidental taxonomist behavior is a particularly influential confounder for PPI, and it is also broadly applicable to any supervised-learning protein dataset.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-026-06491-3.

Introduction

Proteins are ubiquitous macromolecules that drive biochemical reactions, compose biological structures, and enable physiological signaling, all from organized but straightforward physicochemical forces [1–11]. They often accomplish these vital tasks through intricate cross-talk with each other, known as Protein-Protein Interactions (PPIs). In its most basic form, a PPI involves two proteins exhibiting physical contact that mediates chemical or conformational change, especially with non-generic function [12–14]. At scale, paired PPIs can form complex networks and pathways, and are central to life sciences due to their implications in studying fundamental biology, disease, and potential therapeutics. However, traditional characterization of PPIs through in vitro and in vivo methods is expensive and time-consuming. Established higher-throughput methods like Yeast 2-Hybrid (Y2H) can identify thousands of PPIs once libraries are prepared at relatively low costs. More recent methods, such as Cell-Free Two-Hybrid (CF2H), enable the coupling of cell-free protein expression and precise detection capabilities that can correlate to binding affinity [15, 16]. Despite these advances, the scale of all possible human pairwise PPIs (over 200 million possible combinations from approximately 20,000 proteins, only a small fraction of which represent true interactions) necessitates a major improvement in throughput. A comprehensive, computable, and accurate representation of the interactome could serve as the base blueprint of cellular function. To this end, many groups are researching computational methods to predict PPIs from protein identifiers [17, 18]. Some methods use protein or gene IDs to search and extrapolate from established databases; others leverage protein structure solutions or predictions to infer PPIs, and many recent methods leverage Protein Language Models (pLMs) to infer PPIs from primary sequences [14, 19–22]. These platforms, particularly pLMs, offer an attractive solution with the possibility for massively high-throughput screening, amino acid- or atom-level analysis of PPI mechanisms, and inference of functionality [18].

pLMs have shown various levels of claimed success in predicting PPI. However, as with any machine learning model, there are many possible biases in dataset composition that can lead to inflated model performance. PPIs are an interesting machine learning and data compilation problem; while numerous PPI databases curated from the literature contain hundreds of thousands of real PPIs (and millions of likely ones), PPIs are extremely dependent on the biochemical context of the experiment. Factors such as post-transcriptional and post-translational modifications (PTMs), chaperone-mediated folding, protein cofactors, mutagenesis, pH, and many others ensure that recorded PPIs between two entities are rarely guaranteed to be universal true positives. Because of this, we reason that PPI machine learning models trained on general PPI data actually learn the relative plausibility of PPI, without accounting for biochemical context. In other words, we reason that pLM-based methods may infer that there exists a combination of biochemical factors, like PTMs and allosteric changes, that allow for a PPI without guaranteeing that the combination of characteristics ever occurs in reality. Therefore, positive PPI data and prediction come with a high level of ambiguity. Additionally, it is impossible to prove empirically that something cannot happen, so there are no “real” negative PPI examples to contrast against PPI positives. The Negatome project has improved the availability of extremely unlikely PPIs through manual analysis; however, even then, all of the Negatome−2.0 data splits include only 6,000 negative PPI protein pairs when combined [23]. This extreme data imbalance ensures that any binary prediction machine learning model trained on the unmodified available data will almost always predict “positive” [14, 24]. Previous popular negative curation methods leveraged information about protein subcellular compartment localization to estimate negatives, under the logic that proteins that localize to different cellular regions are unlikely to interact [25, 26]. While this logic is reasonable, proteins that do not interact in vivo due to their location may be able to interact in vitro, which is still important information in the coming age of protein design [14, 27]. In our earlier work around SYNTERACT, we posited that pLMs are accidental localizers: pLM features can be used to predict the subcellular compartment localization of proteins from their primary sequence with high accuracy [28–30]. Thus, pLM-based PPI models trained on real positives and subcellular negatives “cheat” and learn to predict the difference in localization, rather than PPI (or in addition to PPI) [14, 31].

More recently, papers by Bernett et al. established a “gold-standard” dataset built with different sequence-similarity-based biases in mind, attempting to control for pLMs memorizing domains or motifs [20]. Their balanced binary PPI dataset was constructed using positives from the Human Integrated Protein-Protein Interaction Reference (HIPPIE), and negatives generated through randomly selecting protein pairs to match the number of positives [20]. This random selection method for PPI negatives is currently the most popular approach, which leverages the assumption that randomly generated or paired amino acid sequences should not induce any meaningful PPI when expressed together, a good assumption when considering the highly specific nature of many PPIs. It also, at least partially, alleviates the accidental localizer issue by ensuring a mix of subcellular localization trends in the negative examples. Therefore, a model cannot purely rely on the localization confounder. Evaluation splits of the Bernett dataset were strategically partitioned so that there were no node degree biases, no sequence overlap (referred to as C3 by Park and Marcotte [32]), and a maximum sequence similarity of 40% between any of the train, validation, and test sequences. Works from Ko et al. and Bernett et al. have benchmarked many established pLM- or neural network-based PPI methods on this gold-standard dataset and have shown that performance is roughly limited to 0.7 AUC and 65% accuracy with current modeling capabilities [21, 22, 33, 34].

Yet, we make a simple observation that on multi-species datasets, unlike Bernett’s gold-standard human-protein-only PPI dataset, pLM-based methods showcase much higher performance metrics on withheld data. Even our SYNTERACT model, trained on a modified version of BioGRID, showcased over 90% accuracy on our withheld test set [14, 35]. In Ko et al.’s paper, we also observe high AUC scores ranging from 0.8 to over 0.9 with multi-species performance (STRING-db v11, C3, 40% CD-HIT) [22]. We believe this trend holds for much of the literature [25, 34, 36–41]. Recent work has also shown that pre-training data leakage from pLMs can inflate PPI scores, though the effect is modest (approximately 3% AUROC) and distinct from the confounders we explore here [42]. Of course, to a certain degree, higher multi-species performance is expected, as more species means more data and sequence diversity, and machine learning models should improve accordingly. However, the performance difference is staggering.

In this paper, we highlight observations and experiments that explore PPI performance discrepancies and put forward a hypothesis for the suspiciously high multi-species PPI performance reported in the literature. Specifically, we expand upon work analyzing the encoded phylogenetic relationships in pLM embeddings by demonstrating that neural networks trained on pLM embeddings can predict the taxonomic origin of protein primary sequences, even at extreme levels of discrimination [43–45]. We further postulate and support that it should be even easier than predicting the precise taxonomic classification of one protein to distinguish whether two proteins share a taxonomic origin. If it is possible for pLM-based methods to model the phylogenetic distance of two protein sequences, it implies that PPI models trained on multi-species data could be inadvertently learning taxonomic or phylogenetic features, and not PPIs. We refer to this hypothesis as the “accidental taxonomist,” where multi-species PPI models exploit the phylogenetic distance of negative examples to artificially inflate their performance. We further support our hypothesis by training a variety of pLM-based models in two conditions, where (1) the negatives are sampled randomly from BioGRID and (2) the negatives are sampled randomly from within the same species of BioGRID data. We show on rigorously split evaluation sets, similar to Bernett’s gold-standard (C3, 40% CD-HIT), that option (1) will “cheat” and option (2) cannot; implying a straightforward solution to this problem through strategic sampling.

Unfortunately, while we uncover the accidental taxonomist for PPI prediction, it is also potentially a key confounder in any supervised protein dataset, where certain classes or sections of labels may belong to a phylogenetic region, thereby compromising the underlying performance of the model despite reasonable evaluation set metrics. We intend for researchers to carefully consider phylogenetic distances and taxonomic biases to establish standardized controls that can be implemented to improve the performance and utility of pLM-based supervised learning.

Results

Multi-species PPI datasets are subject to inherent phylogenetic biases

BioGRID is a curated database of experimentally validated protein interactions containing only positive PPI edges [35]. A common approach for generating negative examples is to randomly pair proteins that are not known to interact. To characterize the taxonomic composition of such randomly generated negatives, we independently shuffled the protein A and protein B columns of BioGRID 100 times, effectively simulating random negative pairing. The resulting randomly paired examples shared the same organism of origin only about 31% of the time on average (Fig. 1). In contrast, nearly all real (positive) PPIs in BioGRID involve two proteins from the same species, since interactions are typically characterized within a single organism. Therefore, assuming a deep learning model could perfectly predict whether two proteins share a taxonomic origin and using the BioGRID ratios as reference, the perfect taxonomic classifier would identify positive examples 96.5% correctly (both proteins are intra-species) and negative examples with 68.2% accuracy (by predicting “negative” whenever it detects an inter-species pair). On a balanced dataset of 100,000 examples, a simple simulation reveals that this taxonomic shortcut alone would yield 82% accuracy, an F1 score of 0.85, an MCC of 0.67, and an AUROC of 0.82. More metrics and associated plots can be found in Supplemental Fig. S1.

Fig. 1.

Taxonomic composition of positive PPI pairs versus randomly generated negatives in BioGRID. A Proportion of intra-species and inter-species pairs among positives (96.5% intra-species) versus shuffled negatives (31.8% ± 0.02% intra-species, mean ± s.d. over 100 shuffles). B Radial bar chart comparing organism pair frequencies (log scale) between positives (blue) and shuffled negatives (orange). Positives are dominated by intra-species pairs (Hs-Hs, Sc-Sc, Ec-Ec), while negatives are dominated by inter-species pairs (Hs-Sc, Ec-Hs). Species abbreviations: Hs = H. sapiens, Sc = S. cerevisiae, Ec = E. coli, Sp = S. pombe, At = A. thaliana, Mm = M. musculus, Dm = D. melanogaster, Ce = C. elegans, SCV2 = SARS-CoV-2

pLM features can be used to predict precise taxonomic origin

We benchmarked several popular pLMs on large protein datasets with labels based on taxonomic discrimination, ranging from domain to species. Datasets were embedded and fed to MLP probes, highlighting the intrinsic correlation between the embeddings and the tasks. We also trained on a similar balanced binary classification dataset where pairs of sequences were either from the same species or not. Performance on these compiled datasets is showcased in Fig. 2.

Fig. 2.

Performance heatmap of all models and datasets evaluated comparing pLM embedding correlations to taxonomic discrimination. Weighted F1 scores are presented for each model-dataset combination, chosen because of the large class imbalance. The different datasets constructed by taxonomy are arranged along the rows, with individual model performance within each column

The random vector column in Fig. 2 shows truly random performance across the test datasets, which varies with the number of classes in each dataset. For multi-class tasks, random vectors yield near-zero weighted F1 because the probe collapses to predicting a single dominant class; for the balanced binary taxonomy task, the weighted F1 of 0.33 reflects the same single-class prediction behavior (predicting one class on a balanced binary dataset yields precision of 0.5, recall of 0.5 for the predicted class and 0.0 for the other, resulting in a weighted F1 of 0.33). A transformer with random weight initialization and one-hot encoding demonstrated basic performance gains from the addition of basic sequence features alone, also known as a homology controls [30]. All other probes were fed embeddings from a pLM that had undergone substantial semi-supervised denoising, which clearly enhanced the feature vectors for taxonomic classification, as evidenced by test F1 scores relative to random and homology controls.

Highlighted in Fig. 2, most pLM embeddings could be used to almost perfectly predict sequences with vast phylogenetic distances at the domain level in taxonomy (corresponding to Archaea, Bacteria, and Eukarya), with the best performer at 0.97 F1 (ProtT5). At the kingdom level, taxonomy, this performance dropped to 0.69 F1 (ESMC), 0.49 F1 (ProtT5) at phylum, 0.36 at order (ESMC), 0.28 at family (ESMC), and 0.21 at genus (ESMC). At the most discriminative level, taxonomy (among 618 species classes), the best-performing pLM embeddings achieved significantly better-than-random performance, with an F1 score of 0.18, compared with 0.00.

While taxonomy and taxonomy are different task formulations (multi-class classification of a single protein vs. binary classification of a protein pair), they probe the same underlying capability: the degree to which pLM embeddings encode phylogenetic information. The key insight is that even when precise species classification is unreliable (0.18 F1), the binary same-vs-different discrimination is highly accurate. The model probes performed much better on taxonomy, identifying if two input sequences share a species or not, compared to precise species classification in taxonomy. Most pLMs exceeded 0.8 F1 on this task, with ESMC embeddings yielding the best F1 score of 0.87. This is directly relevant to PPI prediction, where the model receives two protein sequences and must produce a binary output, mirroring the taxonomy formulation. Because pLMs achieve near-perfect domain-level discrimination (0.97 F1) even in the harder single-protein multi-class setting, paired binary tasks at coarser taxonomic levels would trivially approach perfect performance, making species-level discrimination the most conservative and informative test of the accidental taxonomist hypothesis.

Strategic sampling prevents “cheating” on PPI sets

Having demonstrated that pLM embeddings encode sufficient phylogenetic information to reliably distinguish whether two proteins share a species of origin, we next tested whether this capability actually inflates PPI prediction performance. Specifically, we compared two negative sampling strategies on a multi-species PPI dataset derived from BioGRID to determine whether models exploit phylogenetic distance as a shortcut for PPI classification.

We attempted to rectify the accidental taxonomist behavior by never feeding a “negative” PPI to a model from a different species. Instead, we randomly selected sequence pairs from within the same species to construct negatives; we refer to this approach as Strategic Sampling (SS). This simple change from the Normal Sampling (NS) approach, in which negatives are sampled randomly from the entire multi-species dataset, yielded substantially different performance metrics on the validation and test sets (Fig. 3). The validation sets follow the same sampling as the training set (NS validation for NS models, SS validation for SS models), but both models are evaluated on an SS test set to reveal whether the NS model has learned to cheat. The NS models averaged 0.71 MCC on the NS validation set, compared with 0.39 MCC for the SS models on the SS validation set, indicating that the NS models rely heavily on phylogenetic distance measurements. This is confirmed on the shared SS test set, where the NS model achieved only a 0.23 MCC, compared to 0.34 MCC for the SS models, a significantly lower performance from the NS model, as it is unable to effectively exploit phylogenetic distances on intra-species negatives. Both differences are statistically significant, with a t-test p < 0.0001 for each (n = 5). Notably, the 0.23 MCC score from NS is much better than random performance (equivalent to 62% accuracy, 0.61 F1 score, and 0.63 AUROC), meaning that the NS models do not solely rely on phylogenetic distances and appear to still learn aspects of PPI. Excitingly, the best test SS MCC of 0.37 is approximately 23% higher than the current best MCC score on Bernett’s single-species dataset (0.30 MCC). While these datasets differ in their underlying data sources (BioGRID vs. HIPPIE), approximately 45% of Bernett’s positive pairs overlap with BioGRID, and both employ the same evaluation rigor (C3 splitting, 40% sequence identity clustering). This comparison is therefore suggestive, though not definitive, that incorporating multi-species data with proper phylogenetic controls is an effective way to enhance PPI prediction performance and generalizability.

Fig. 3.

Comparison of SS (blue) and NS (orange) approaches with Violin plots of MCC scores on the validation and test sets over five PPI training runs. The five NS and SS score distribution differences are highly statistically significant, with NS greatly outperforming SS on the validation set and underperforming on the test set

Training dynamics reveal distinct reward hacking

We analyzed the training dynamics of the NS and SS training runs by tracking the training loss, average predicted probability of PPI for positive examples, and average predicted probability of PPI for negative examples. After calculating a time-weighted exponential moving average and the 99.99% confidence interval (Fig. 4), these data reveal that the training trajectories of NS and SS are completely disjoint.

Fig. 4.

Comparison of training dynamics between SS (blue) and NS (orange) across five training runs. Left: Cross-entropy training loss over training steps, with the random baseline (ln 2 ≈ 0.693) shown as a dotted line. Middle: Average predicted probability of PPI for positive examples, showing NS models drive predictions to 0.94 while SS models plateau at 0.75. Right: Average predicted probability for negative examples, where both approaches converge to approximately 0.5. Shaded regions represent 99.99% confidence intervals; faint black lines show individual seed trajectories. The divergent positive prediction trajectories reveal reward hacking behavior in NS models

In particular, the average positive PPI probability approaches 0.94 for NS compared to 0.75 for SS, presenting clear reward hacking behavior [46], as the loss was driven extremely low by increasing the associated logit when the NS models measure a zero phylogenetic distance. Interestingly, the average negative PPI prediction probabilities are remarkably similar, approximately 0.5, implying that negatives are much harder for PPI models to classify for NS and SS models.

Inter-species interactions are invisible regardless of sampling

A natural question arising from the accidental taxonomist hypothesis is what happens to inter-species PPI prediction. To address this, we trained SS and NS models on BioGRID Multi Validated (BioGRID-MV) (version 5.0.253) [35], a subsection of BioGRID where all examples have been experimentally validated by at least two methods from at least two sources. BioGRID-MV also contains a significant amount of inter-species positives, 35,970 ( 7%) of the 531k total pairs. We trained new NS and SS models and evaluated both models on a shared test set containing inter-species positives and intra-species positives with SS negatives.

The test set contained 2,815 examples (2,762 positives, 53 negatives), including 1,091 inter-species positives dominated by human-SARS-CoV-2 pairs (982 of 1,091). The extreme positive-to-negative ratio reflects that inter-species PPIs in BioGRID are predominantly viral-host interactions, for which few intra-species (SS) negatives can be generated from the limited set of viral proteins. On the full test set, the SS model achieved an AUROC of 0.80 (95% CI: 0.73-−0.86) compared to 0.62 (95% CI: 0.56-−0.67) for the NS model (DeLong test: Z = 3.60, ; Fig. 5).

Fig. 5.

ROC curves comparing SS (blue) and NS (orange) models on the full BIOGRID-MV inter-species experiment test set (). Shaded regions indicate 95% bootstrap confidence intervals. The SS model achieves significantly higher discriminative ability (AUROC = 0.80) than the NS model (AUROC = 0.62; DeLong ), consistent with the accidental taxonomist effect inflating NS validation performance while degrading generalization

The inter-species subgroup reveals the most striking result: the SS model classified zero of 1,091 inter-species positives as positive interactions (0% recall), and the NS model classified only two (0.2% recall). Both models assigned inter-species positives a mean predicted probability near 0.46, barely distinguishable from a coin flip, indicating that inter-species pairs are effectively invisible to both sampling strategies. On intra-species positives, the SS model substantially outperformed the NS model (AUROC 0.81 vs. 0.49; DeLong Z = 11.37, ; Supplemental Fig. S4), consistent with our earlier findings.

Discussion

Computational PPI prediction is a vital problem due to its widespread applicability across life science domains. With performance plateauing around 65% accuracy and 0.3 MCC on rigorously constructed single-species datasets [20, 21], the strikingly higher performance reported on multi-species datasets demands explanation. As we demonstrated, the taxonomic composition of randomly generated negatives in multi-species datasets such as BioGRID creates a shortcut that closely aligns with reported multi-species results from the literature [22, 25, 34, 37–41]. It is worth noting that BioGRID contains over 44% human data, with other large percentages coming from model organisms. Much larger and more diverse repositories, like STRING-db, would exacerbate this issue further, with an even smaller fraction of protein pairs coming from the same species if the entire dataset were shuffled.

To explore the capabilities of pLM-based methods to distinguish the taxonomic origin of primary sequences, we expanded upon existing work [43, 44] on pLM embeddings and encoded phylogenetic distances to characterize how well various neural network probes can classify taxonomic origin across a wide range of discrimination, from broad domain to species. Unsurprisingly, the best-performing model on average, ESMC [47, 48], achieved almost perfect performance in predicting sequences with vast phylogenetic distances at the domain level in taxonomy (corresponding to Archaea, Bacteria, and Eukarya). We observed a gradual decline in overall performance as the phylogenetic distance of the labels decreased. Interestingly, at the most discriminative label set in taxonomy, among 618 species classes, ESMC maintained a much better-than-random performance of 0.18 F1 score (vs. 0.00 random), and sometimes the exact species of origin could be determined from only a single primary sequence. Considering the training dataset utilized sequences that were maximally 40% similar to the validation and test sets, we were surprised to achieve better-than-random performance at the species level and encourage future work to explore which specific combination of learned features from semi-supervised denoising drives this performance. Nevertheless, despite being better than random, a 0.18 F1 score is far from reliable performance.

However, consistent with our hypothesis, we demonstrated that when fed two sequences in taxonomy, aligned with the standard PPI-prediction workflow, neural networks could effectively predict whether each sequence came from a different species. In fact, most pLMs tested exceeded 0.8 F1 score on this balanced binary prediction task, with ESMC achieving 0.87 F1. Even the homology controls exceed 0.7 F1. Thus, although neural networks cannot predict the exact species from pLM embeddings, they can reliably detect whether two sequences have a nonzero phylogenetic distance. We refer to this finding as the accidental taxonomist, where pLM-based methods exploit phylogenetic distance distributions across classes to inflate their performance. We reasoned that this should be particularly applicable to PPI datasets, where typical datasets have pairs labeled “1” with no phylogenetic distance and most pairs labeled “0” with some distance, and confirmed this by comparing the performance of NS- and SS-trained models. In particular, the reward hacking associated with accidental taxonomist behavior is clearly evident in the evaluation metrics in Fig. 3 and in the training loss and logit distributions shown in Fig. 4.

We acknowledge that SS likely introduces slightly higher label noise than NS, as two random proteins from the same species are more likely to interact (but remain uncatalogued) than two proteins from different species. However, the magnitude of the performance collapse, from  0.7 MCC to  0.2 MCC on the SS test set, is too drastic to be explained by label noise alone. If label noise were the primary driver, we would expect the SS-trained models to suffer similarly, yet they maintain consistent performance across validation and test sets. This asymmetry strongly supports the accidental taxonomist hypothesis. We also note that, although we employed a custom transformer-based architecture rather than simpler alternatives to ensure our comparison was representative of state-of-the-art methods, the accidental taxonomist is fundamentally a data issue rather than a model issue; we expect similar NS/SS discrepancies with any reasonable architecture.

Importantly, despite comparable dataset construction rigor, our multi-species SS PPI models all exceed the 0.3 MCC plateau for single-species PPI, suggesting that adding sequence diversity may improve PPI performance and generalizability when properly controlled. For fairness, we note that our multi-species dataset is substantially larger than typical single-species datasets, and part of this improvement could be attributable to scale rather than phylogenetic diversity alone. Future work with size-matched controls would be needed to fully disentangle these effects, but for our purposes it is clear that the training loss curves between NS and SS have exceedingly different trends early into training, well within the size of typical training datasets and runs. More reported evaluation metrics from our runs are shown in Supplemental Figs. S2 and S3.

From this work, our previous identification of accidental localizers, and the outstanding contributions of Bernett, Ko, Rost, and many others, exploring the flaws and biases behind PPI prediction, we can safely assume that very high PPI performance metrics (for example above 0.95 AUC, 0.8 MCC, 0.9 F1, 90% accuracy, etc.) are likely due to some form of data leakage or underlying confounders, in addition to (or instead of) genuine learning of PPI. High-throughput computational PPI remains an unsolved problem, and dataset construction and curation challenges remain major obstacles.

Our findings also have direct implications for host-pathogen PPI prediction. Our inter-species experiment on BioGRID-MV confirmed that both NS and SS models exhibit near-zero recall on inter-species interactions (Fig. 5): neither model correctly identified more than 0.2% of inter-species positives, which were dominated by human-SARS-CoV-2 pairs (supported by [42]). This represents a significant blind spot, as viral entry mechanisms and bacterial effector-host protein interactions are inter-species PPIs of great biomedical interest. Importantly, even the SS model, which cannot exploit phylogenetic distance, fails on inter-species positives, suggesting that dedicated inter-species training data and evaluation protocols may be necessary for host-pathogen applications.

Unfortunately, our analysis underscores a more fundamental flaw in pLM dataset compilation beyond PPI prediction. For example, one common application of pLM embeddings is predicting Enzyme Commission (EC) numbers, a categorization scheme for different enzymatic reactions. If a documented EC number occurs only in a specific taxonomic family, a model trained on such data may inadvertently learn to classify by taxonomy rather than by enzymatic function. This same principle applies to any supervised protein-learning task, including regression, where a specific label range could belong exclusively to a taxonomic range.

It is conceptually intuitive to imagine an attention-based neural network exploiting confounders during multiple-input tasks, such as PPI, as these models directly compare two inputs via dot product [49]. However, even simpler neural networks and single-protein input datasets are not immune to the accidental taxonomist effect. Neural networks are capable of storing various training distributions into their weights via gradient descent [50–52]. For instance, a simple feed-forward network could organize one layer of weights to correspond to the average taxonomic features for each class, causing inputs with specific matching taxonomic features to produce higher values after this layer.

Additionally, we suspect that genomic language models, leveraging DNA or codon vocabularies to study proteins, will be even more susceptible to the accidental taxonomist phenomenon than amino acid–based pLMs. Codon usage bias is a powerful feature for predicting taxonomic origin on its own [45, 53], adding another layer of exploitable signal on top of the amino acid-level features already present in pLM embeddings.

Finally, there may be other “accidental _______” phenomena where unintended physical, chemical, biological, or even social trends, surrounding which proteins researchers choose to study, distract machine learning models from learning the underlying categorizations and principles of interest. We hope to see future work exploring how neural networks “cheat” on protein datasets and how improved dataset design and controls can mitigate these confounders.

Methods

Prototyping phylogenetic bias impact

The February 2025 release of BioGRID (4.4.243) was downloaded (BIOGRID-ALL−4.4.243.tab3.zip), and protein identifiers were mapped to UniProt accessions, with Swiss-Prot prioritized. IDs were mapped to sequences using the UniProt ID mapper tool https://www.uniprot.org/id-mapping and the primary sequence and organism of origin were recorded for each protein pair [35, 54]. The six columns in our dataset include each UniProt ID (A, B), each sequence (SeqA, SeqB), and each organism (OrgA, OrgB). To characterize the taxonomic composition of randomly generated negatives, we independently shuffled the protein A and protein B columns (and their associated organism labels) 100 times, which simulates the random protein pairing commonly used for negative generation. For each shuffle, we recorded the fraction of randomly paired proteins that shared the same organism of origin, as shown in Fig. 1. This analysis quantifies what fraction of random pairings would be intra-species vs. inter-species, revealing the taxonomic imbalance that models can exploit if a dataset used uncontrolled random sequence pairing to generate PPI negatives. We simulated the classification metrics for a perfect OrgA = OrgB vs. OrgA ≠ OrgB classifier by building random predictions and labels. The 100,000 data points consisted of 50% labeled one and 50% labeled zero, where 31% of the zeros were false negatives per the results of the BioGRID shuffling. The actual predicted probabilities were randomly generated from a uniform distribution between 0.51 and 1.00 for true positives, 0.00 to 0.49 for true negatives, and 0.51 to 1.00 for false negatives, allowing for the estimation of the ROC AUC and PR AUC. Accuracy, precision, recall, F1, and MCC were also calculated (Supplemental Fig. S1).

Probing taxonomic features

We constructed two types of taxonomic probing tasks. The first type (taxonomy through taxonomy) is a single-protein multi-class classification task: given one protein sequence, predict its taxonomic label. The second type (taxonomy) is a paired-protein binary classification task: given two protein sequences, predict whether they originate from the same species or not (i.e., whether protein A comes from species X and protein B comes from species Y, where X ≠ Y). The paired formulation directly mirrors the input structure of PPI prediction.

For the single-protein tasks, datasets at different taxonomic discrimination levels were constructed from Swiss-Prot, downloaded on 7/22/2025 (release 2025_03) [54]. We kept the taxonomic lineage column to build labels and retained sequences of lengths greater than 20 and less than 2048. Sequences were deduplicated and clustered at 40% sequence identity with a word size of two using CD-HIT [55], and clusters were randomly assigned to validation and test splits until their count exceeded 10,000 entries, respectively. For each taxonomic discrimination, we only kept labels that had at least 100 examples total, resulting in three domain, 17 kingdom, 68 phylum, 126 class, 248 order, 371 family, 545 genus, and 618 species classes. The total training set sizes consisted of 444k domain, 450k kingdom, 458k phylum, 451k class, 446k order, 434k family, 419k genus, and 265k species examples. We enforced that there was no sequence or cluster overlap between any splits to ensure evaluation rigor. The datasets are referred to as taxonomy, taxonomy, taxonomy, taxonomy, taxonomy, taxonomy, taxonomy, and taxonomy throughout the work for clarity of the discrimination level.

The taxonomy dataset was compiled by taking taxonomy and randomly constructing protein pairs from within each data split. 50,000 random sequences from the same species were sampled for the training split, 500 for validation, and 500 for testing. Then, the dataset was balanced by randomly selecting sequences from different species. The final dataset, comprising 100,000 training, 1000 validation, and 1000 test sequences, presented a binary prediction problem: determining if two sequences had any phylogenetic distance between them, where protein pairs were labeled as zero if they were from the same species and one otherwise.

Models and datasets were probed using the Protify package, an open source project aimed at standardized, effective, and high-throughput analyses with chemical language models [56, 57]. We embedded all datasets using mean and variance pooling on the last hidden state of various pLMs, resulting in a consistently sized vector for each protein describing the distribution of the models’ last hidden state. For single-protein tasks (taxonomy through taxonomy), each example consisted of one pooled vector with o equal to the number of taxonomic classes. For the paired-protein task (taxonomy), each example consisted of two pooled vectors stacked together, with o = 2 for binary classification. We chose many popular pLMs for analysis, including ESM2 eight million parameter version (ESM2), ESM2, ESM2, ESM2, ESM2 three billion parameter version (ESM2), ESMC, ESMC, DSM, ProtBERT (big fantastic dataset version), ProtT5-enc (ProtT5), ANKH-Base, and ANKH-Large [47, 57–60]. For the T5-based models, ANKH and ProtT5, only the encoders were used for embedding generation. Three controls were also used: random vectors, which embed each protein with a randomly generated vector (); random transformer, which embeds proteins through a re-initialized equivalent of ESM2; and OneHot protein, which replaces the pLM embeddings with one-hot encoded strings [57].

Within Protify, MLP probes were trained with the architecture:

where denotes Layer Normalization, mapping input (with d being the embedding dimensionality of the pLM) to output where o was the number of classes in the dataset. , , , , b corresponded to the associated bias term in the linear layer, σ was ReLU [61], h was 8,192, p was calculated with

and a dropout of 20% was applied before every MatMul, except the first one [62]. We optimized cross-entropy loss with a batch size of 64, learning rate of , AdamW optimizer, weight decay of 0.01, and evaluation on the validation set until a patience of five was exceeded, comparing the validation loss. The best weights were kept and evaluated on the test set [56].

Strategic sampling experiment

The protein sequences in BioGRID 4.4.243 were deduplicated and clustered at 40% sequence identity with a word size of two using CD-HIT [55]. Clusters were added to a list until their paired sequences exceeded 5,000 BioGRID examples for validation and testing, respectively. We ensured that every data split was disjoint in terms of sequence and cluster ID, following the C3 strategy from Park and Marcotte [32]. Negatives were generated within each data split by either sampling random proteins from the clusters (Normal Sampling / NS) or sampling random proteins from the clusters that are from the same organism (Strategic Sampling / SS). Each candidate negative pair was cross-referenced against the full set of known positive interactions; if a candidate pair was found among the positives, it was discarded and resampled to avoid false negatives. The final datasets for NS and SS both consisted of 4,523,432 training examples, 10,070 validation examples, and 10,034 test examples, with a perfectly balanced label distribution; the organism distributions between the positive and negative examples were approximately the same, and the datasets used a total of 76,045 unique protein sequences.

We trained a hierarchical transformer neural network (Fig. 6) to map pLM embeddings to binary logits. The embedding model of choice was ESMC [47, 48], which we froze and extracted the last hidden state for each sequence, truncating it to a length of 512 to minimize computational expense. This followed recent findings that larger maximum lengths have a minimal effect on PPI performance [42]. Because these are not intended as production models and are used for comparison between NS and SS, this truncation was acceptable.

Fig. 6.

Architecture of the PPI prediction model. Two protein sequences are embedded by a frozen ESMC protein language model, producing per-residue embeddings. Each embedding passes through a separate encoder track (non-Siamese) consisting of a linear projection (1152 → 512), a transformer block with SwiGLU feed-forward networks, and a cross-attention pooler that compresses variable-length sequences into 32 fixed tokens . The pooled representations are concatenated () and processed by four transformer blocks with progressive dimensionality reduction () via interleaved projection layers (LayerNorm → Linear → GELU → Dropout). The final representation is mean-pooled and passed to a linear classification head. Input orientation (A, B) is randomly swapped with 50% probability during training

For each sequence A and B, there was a separate encoder track (not shared-weights / siamese network) consisting of a linear layer projection from ESMC hidden size of 1,152 to 512, one transformer block with hidden size 512, and then token-parameter cross-attention, which pooled every protein sequence into consistently sized matrices , [63, 64]. During training, the input orientation (A, B) was randomly swapped with 50% probability to prevent order-dependent biases; the separate encoder tracks exist to give each protein its own representational subspace, which we believe is particularly important for homodimer modeling. Averaging logits from both orientations of the independent tracks at inference is a technique we use in local experiments to marginally improve performance but did not significantly affect results in our experiments. A and B were then concatenated into , which served as the input to the rest of the model. We suspect it is better to concatenate the protein representations instead of combining elementwise, which is popular in the literature [36, 39], enabling the final layers the ability to leverage independent information from both entities. The remainder of the model consisted of four transformer blocks, interleaved with linear layers, which reduced the dimensionality from to , projecting down by a factor of two at every transformer block. The final matrix was mean-pooled into a vector and fed to a final linear layer to build the logits . An expansion ratio of , rounded to the nearest multiple of 256, was used for all transformer block MLPs and all linear layers (aside from attention layers) used a dropout of 0.1. Binary cross-entropy was employed alongside the AdamW optimizer [65], a batch size of 128, learning rate of , evaluation on the validation set every 5000 steps, and all models were trained for one epoch on a single GH200 GPU; the weights with the best validation MCC were kept and test scores were reported.

In case each model curated slightly different decision boundaries, and different thresholds were optimal to split logits into positives and negatives, we split the precision-recall curve into 100 discrete points and tried each threshold. The threshold used for the metrics calculations was chosen based on which one produced the maximal F1 score. Technically, macro F1 was used for threshold selection and weighted averages were reported for F1, precision, and recall (Supplemental Fig. S3); however, because the datasets were balanced, this did not impact the final result.

Inter-species positives experiment

To evaluate model performance on inter-species PPIs, we used BIOGRID-MV version 5.0.253, a curated subset of BioGRID enriched for multi-species and viral-host interactions [35]. Protein sequences were clustered at 40% identity with 80% coverage using MMseqs2 [66]. An inter-species-aware C3 split was constructed by greedily assigning clusters to validation and test sets, prioritizing clusters containing inter-species positive pairs (where protein A and protein B originate from different species) until each split contained at least 500 total pairs and 100 inter-species positives. The remaining clusters formed the training set. Protein and cluster disjointness was enforced across all splits. Negatives were generated using the same procedure as the main experiment: SS negatives from within the same species and NS negatives from random pairing. Both models shared an identical SS test set to enable direct comparison. The same model architecture and hyperparameters described above were used. ROC curves were compared using the DeLong test for paired AUROCs, with 95% confidence intervals computed via the DeLong method and bootstrap confidence bands (2,000 replicates) visualized on the ROC plots.

Author contributions

Conceived and designed the experiments (LH, JPG), performed the experiments (LH, TP, NR), analyzed the data (LH, TP, NR), contributed materials/analysis tools (LH, JPG), wrote the paper (LH, JPG), edited and reviewed the manuscript (LH, TP, NR, JPG).

Funding

This work was partly supported by the University of Delaware Graduate College through the Unidel Distinguished Graduate Scholar Award (LH), the National Science Foundation through NAIRR pilot 240064 (JPG), and the National Institutes of Health through NIGMS T32GM142603 (LH, NR), R01HL178817 (JPG), R01HL133163 (JPG), and R01HL145147 (JPG).

Data availability

Datasets are available at https://github.com/Gleghorn-Lab/PLMConfounders. Protify and its associated datasets are accessed at https://github.com/Gleghorn-Lab/Protify and on Hugging Face at https://huggingface.co/collections/GleghornLab/protify. The taxonomic classification datasets are available on Hugging Face: Domain (https://huggingface.co/datasets/GleghornLab/taxonomy_domain_0.4_clusters), Kingdom (https://huggingface.co/datasets/GleghornLab/taxonomy_kingdom_0.4_clusters), Phylum (https://huggingface.co/datasets/GleghornLab/taxonomy_phylum_0.4_clusters), Class (https://huggingface.co/datasets/GleghornLab/taxonomy_class_0.4_clusters), Order (https://huggingface.co/datasets/GleghornLab/taxonomy_order_0.4_clusters), Family (https://huggingface.co/datasets/GleghornLab/taxonomy_family_0.4_clusters), Genus (https://huggingface.co/datasets/GleghornLab/taxonomy_genus_0.4_clusters), Species (https://huggingface.co/datasets/GleghornLab/taxonomy_species_0.4_clusters), and Different (https://huggingface.co/datasets/GleghornLab/diff_phylo). The sequences and taxonomic origin were originally gathered from UniProt release 2025_03. PPI training, validation, and test datasets are compiled automatically at training time; the exact examples used in the study can be found at https://github.com/Gleghorn-Lab/PLMConfounders/tree/main/processed_datasets, sourced from BioGRID version 4.4.243. The BioGRID IDs, mapped to UniProt IDs and sequences, are also hosted on Hugging Face: https://huggingface.co/datasets/Synthyra/BIOGRID. Selected code is available at https://github.com/Gleghorn-Lab/PLMConfounders. Protify and its code can be accessed here: https://github.com/Gleghorn-Lab/Protify.

Code availability

Selected code is available at https://github.com/Gleghorn-Lab/PLMConfounders. Protify and its code can be accessed here: https://github.com/Gleghorn-Lab/Protify.

Declarations

Ethics approval and consent to participate

NA

Consent for publication

Not applicable.

Competing interests

LH and JPG are co-founders of and have an equity stake in Synthyra, PBLLC.