ToxiCompass: an evidence-tracked benchmark for short-peptide toxicity screening and confidence-aware assay prioritization

Abstract

Binary peptide-toxicity predictors support early screening but do not indicate which assays should follow a toxin-like prediction. We present ToxiCompass, an evidence-tracked benchmark and two-stage protocol for short-peptide toxicity screening and assay-family prioritization. The benchmark contains 6114 peptides reconstructed from online database of biotoxins and reviewed Universal Protein Knowledgebase records, including curated toxin-positive peptides and length-matched operational comparators. Among 2866 Stage 2 candidate toxin-positive peptides, 2858 retained at least one observed-positive target/effect-family label after evidence review. Exact-sequence merging, provenance tracking, Many-against-Many sequence searching 2 clustering at 90% sequence identity (MMSeqs90) cluster-level splitting, and explicit evidence rules define the evaluation. Stage 1 assigns a toxin-like score relative to the operational-comparator population, whereas Stage 2 ranks 13 broad target/effect families to prioritize follow-up assays for toxin-positive peptides with retained evidence. On the primary MMSeqs90 cluster split, Stage 1 achieved an area under the receiver operating characteristic curve of 0.9871 and an area under the precision-recall curve of 0.9867, while Stage 2 achieved a Hit@3 of 0.8955. In a more conservative grouped-split sensitivity analysis, Stage 2 Hit@3 was 0.6506. Together, these results support ToxiCompass as a reproducible benchmark linking toxin-like peptide screening with evidence-aware prioritization of follow-up assay families.

Introduction

Short peptides are widely used in antimicrobial discovery, venom-derived pharmacology, drug optimization, and functional engineering [1]. The same activities that make them useful, including membrane disruption, ion-channel modulation, coagulation effects, and receptor engagement, can also produce toxicity in a context-dependent manner [2]. Experimental assays remain indispensable, but they are costly, condition dependent, and difficult to scale to modern discovery and design pipelines [3]. Computational methods can therefore support early triage; for experimental follow-up, however, researchers also need guidance on which broad target/effect families should be examined first.

Early peptide-toxicity predictors established sequence-based binary screening. ClanTox, a classifier of short animal toxins[4], was an early sequence-based predictor. The ToxinPred series comprised in silico methods for predicting peptide and protein toxicity. These methods combined sequence, motif, physicochemical, and machine-learning features and were subsequently updated for both proteins and peptides [3, 5, 6]. Later approaches expanded the available representations. ToxIBTL, a transfer-learning model with an information bottleneck, was introduced for peptide-toxicity prediction [7]. VISH-Pred, an ensemble based on fine-tuned Evolutionary Scale Modeling (ESM) models, provided another protein-language-model-based approach [8]. Recent peptide-specific models include ToxMSRC, which uses a multi-scale residual architecture [9]; PLPTP, which uses motif-interpretable protein-language-model features [10]; the protein-language-model descriptor-fusion model HyPepTox-Fuse [11]; and ToxPLTC, which combines pretrained T5 representations with a text-convolutional network [12].

Structure-aware methods extend this progression by incorporating geometric context. ToxGIN, a graph-isomorphism-network-based peptide toxicity predictor, combines peptide sequence and predicted structure through graph isomorphism networks [13]; PeptiTox, a deep learning framework integrating protein language models and geometric deep learning for peptide toxicity prediction, integrates protein-language-model embeddings, predicted structures, and geometric deep learning [14]; StrucToxNet, a structure-aware peptide toxicity prediction framework integrating a pretrained protein language model with an equivariant graph neural network, pairs pretrained language models with equivariant graph neural networks [15]; and PTP-SMGCA, a dual-modal peptide toxicity prediction framework based on peptide-sequence and molecular-graph fusion with bidirectional cross-attention, fuses peptide sequence and molecular graphs through cross-attention [16]. Related studies have also explored broader protein and peptide toxicity prediction using channel attention, convolutional networks, and gated recurrent units [17], as well as machine-learning approaches to carcinogenicity prediction in drug development [18]. These studies broaden the methodological context, although their endpoints, datasets, and evaluation settings differ from the two-stage task considered here.

Despite this progress, most predictors return a binary decision or broad class label rather than an experimentally oriented shortlist. Toxic peptides can act through channels, receptors, membranes, coagulation pathways, and wider cellular responses. MultiToxPred, an ensemble machine-learning tool for multiclass protein-toxin prediction, predicts 27 protein-toxin classes but uses randomly generated non-toxin sequences [19], whereas ToxPre-2L, a two-level predictor based on multi-view tensor learning and latent semantic learning for peptide toxicity and multifunctional-type prediction, jointly models peptide toxicity and co-functional therapeutic labels [20]. These objectives differ from ranking source-supported broad target/effect families for short toxin-like peptides. Such a framework should also support manual review when the available evidence or model support is limited.

This task requires an evidence-aware benchmark because assay-family annotations are incomplete and unobserved labels cannot be assumed to represent biological negatives. We therefore reconstructed records from online database of biotoxins (BioTD) [21] and reviewed Universal Protein Knowledgebase (UniProt) entries [22] through exact-sequence merging, length matching, Many-against-Many sequence searching 2 (MMSeqs2) clustering at 90% sequence identity (MMSeqs90) [23] cluster-level partitioning, and evidence reconciliation. Stage 1 evaluates toxin-like screening against a defined operational-comparator population. Stage 2 ranks 13 broad target/effect families only for toxin-positive peptides with retained source evidence.

Building on this design, ToxiCompass provides an evidence-tracked benchmark and two-stage evaluation protocol that connects toxin-like screening with evidence-aware assay-family prioritization. The Lite and Full models serve as reference implementations for examining the practical trade-offs between sequence-based and multimodal prediction. Together, the benchmark and these implementations provide a reproducible basis for method comparison while keeping the provenance, supervision, and confidence boundaries of both tasks explicit.

Materials and methods

Study design and task definition

ToxiCompass separates toxin-like screening from follow-up assay prioritization. Stage 1 estimates toxin-like status relative to operational comparators; Stage 2 ranks 13 broad target/effect families for toxin-positive peptides with retained source evidence.

For peptide i, Stage 1 outputs a toxin-like probability . Stage 2 outputs for target/effect-family label k=1,…, K, where K=13. Target and effect families share one routing ontology because the output prioritizes assay families rather than asserting mechanisms. The stages are linked operationally but separately supervised: binary labels define Stage 1, whereas retained target/effect evidence defines Stage 2.

Dataset reconstruction and evidence hierarchy

The benchmark was reconstructed from BioTD and reviewed UniProt records. Sequences were upper-cased, stripped of non-sequence characters, restricted to the 20 standard amino acids and 4–50 residues, and merged by exact sequence. Provenance was retained for audit and source-stratified reporting, not used as model input.

BioTD strict/conservative toxin records and reviewed UniProt toxin annotations supplied positives, with toxin status separated from target/effect supervision. Stage 1 required curated toxin relevance. Stage 2 retained a family label only when reconciled BioTD annotations, linked primary records, or literature-backed reviewed UniProt descriptions provided explicit support; broad fields, domains, Gene Ontology terms, family membership, or organism context alone were insufficient. Peptides with toxin evidence but ambiguous target/effect evidence remained Stage 1 positives with Stage 2 labels masked.

Compatible source descriptions were merged into a multi-label record; unresolved conflicts retained the binary toxin label but excluded the disputed Stage 2 label. Supplementary Methods S2 and Supplementary Table S12 give the full masking and conflict rules.

Stage 1 comparators were reviewed UniProt non-toxin/non-antimicrobial-peptide candidates passing the same filters, exact-conflict removal, and positive-matched length-bin sampling. They are operational benchmark comparators, not proof of universal non-toxicity. External panels were overlap-audited but not used for training (Supplementary Tables S13 and S14).

The final benchmark supported two related tasks. Stage 1 used all curated positives and length-matched operational comparators for binary toxicity screening, with labels . Stage 2 mapped only positives with retained evidence to a 13D target/effect-family matrix with incomplete label evidence. Unobserved entries are not exhaustive biological negatives. Out of 3057 Stage 1 positives, 2866 formed the submitted Stage 2 candidate cohort and 191 remained Stage 1-only. After audit, 2858 candidates retained at least one supported-positive label and eight were all-UNKNOWN. Comparators were excluded from Stage 2 rather than encoded as all-zero vectors.

Primary splits followed exact-sequence merging and MMSeqs2 [23] 90%-identity cluster assignment. A stricter sensitivity analysis additionally joined available source-accession, toxin-family, and explicit parent/parent-analog groups before assignment. No records met the prespecified verified-precursor grouping criterion. Table 1 summarizes the submitted split. The sensitivity split yielded Stage 1 4891/612/611 and inherited Stage 2 2410/112/344 partitions. An exhaustive post hoc audit compared all 1223 validation/test peptides with each of the 4891 training peptides. Maximum similarity was operationalized as the highest MMseqs2 local-alignment score; a complementary estimand reported the highest exact identity among hits with bilateral coverage ≥0.80. The audit identified 55 guard-qualified alignments involving 19 evaluation peptides (11 validation and 8 test), indicating residual sequence-overlap risk. These model results are therefore treated as a sensitivity analysis rather than a fully guard-clean estimate. Supplementary Methods S1–S6, Supplementary Tables S1–S16 and S32, and Supplementary Data S1 provide the rules and peptide-level audit.

Table 1.

Dataset composition and split-integrity summary.

Item	Result
Stage 1 curated positives	3057
Stage 1 operational comparators	3057
Stage 1 total samples	6114
Stage 1 train/validation/test	4890/612/612
Stage 2 candidate positive rows (original submission split)	2866
Rows retaining ≥1 final-audited observed-positive label	2858
All-UNKNOWN after final audit	8
Positives used only for Stage 1	191
Stage 2 train/validation/test	2292/287/287
MMSeqs90 clusters	4215
Cross-split exact overlap	0
Shared MMSeqs90 clusters across splits	0
Mixed toxin-positive/operational-comparator MMSeqs90 clusters	0

Leakage control here refers specifically to benchmark construction and supervised train/validation/test partitioning under the exact-sequence/MMSeqs90 protocol. It does not imply that evaluation peptides or their homologs were absent from upstream model-training sources or sequence databases queried during inference. Evolutionary Scale Modeling 2 (ESM2) public source-release overlap, Protenix-v2 structural-source overlap, and inference-time multiple sequence alignment (MSA) and Protein Data Bank (PDB) exposure are reported separately in Supplementary Methods S6.5 and Supplementary Table S47 (Supplementary Results S14).

Input representations and predicted-structure hypotheses

Lite uses mean-pooled facebook/esm2_t30_150M_UR50D residue embeddings (excluding padding and special tokens), with one Stage 1 and 13 label-wise Stage 2 Light Gradient Boosting Machine (LightGBM) classifiers. Amino acid composition (AAC)/dipeptide composition (DPC) and physicochemical LightGBM models are separate sequence baselines. Full adds MSA, predicted-structure, structure-quality, and physicochemical inputs for multimodal review.

Full represents each peptide with six modality tokens:

The sequence token summarizes residue features; ESM2 and MSA tokens encode pooled language-model and profile information [24]; and the physicochemical token forms a separate lightweight stream. Structure and quality tokens summarize five Protenix-v2 [25] predictions for the same peptide as alternative hypotheses, not independent samples.

The five structure hypotheses are

Each predicted structure becomes a residue graph encoded by a shared geometric vector perceptron [26]:

Quality-aware aggregation yields structure and quality tokens:

Here, and denote scalar and vector features for residue v, denotes geometric edge features, denotes a quality-aware seed weight, and summarizes confidence and variability across the five predicted structures. Low-quality or variable hypotheses receive lower confidence. Predicted structures remain screening features, not experimental evidence.

Figure 1 summarizes the task, inputs, and reference implementations; quantitative Lite/Full comparisons appear in Results.

Figure 1.

ToxiCompass task hierarchy and reference implementations; (a) Stage 1 contrasts curated toxin positives with operational comparators; target-supervised positives were evaluated separately in Stage 2 ranking across 13 target/effect families; the evidence states summarize the three-state audit, whereas the submitted checkpoint used the closed-world objective in section Submitted closed-world Stage 2 objective and three-state evidence audit; (b) full-mode input views; (c) sequence/ESM2 lite workflow; (d) multimodal full workflow with five predicted-structure seeds and task-specific fusion.

Target-aware multimodal architecture

The Full model fuses the six modality tokens with a task-specific multi-gate mixture-of-experts (MMoE) [27]. The expert set contains modality-specific experts and a joint expert over the concatenated tokens, and separate gates are learned for binary screening, target/effect-family ranking, and confidence diagnostics:

Here, denotes expert e, E is the number of experts, and is the gate parameter for task t. The separate gates allow different prediction branches to assign different mixture weights to the same modality set.

Stage 1 applies a binary head to the binary task representation:

Stage 2 uses a label-conditioned head. For each target/effect-family label k, a learnable label query attends to the target-task representation, the six modality tokens, and the five seed-level structure embeddings:

The final Stage 2 logit combines the target-task representation, label-conditioned context, a gated structural residual, and a low-weight seed-level residual:

Here, is the label-specific structural residual, is the learned structure gate for label k, controls the contribution of predicted-structure features under label context and structure quality, and η controls the seed-level residual contribution. Together, these terms permit quality- and label-dependent use of predicted structures.

No fixed predicted local distance difference test (pLDDT) or global predicted distance error threshold discards a prediction at inference. Quality features continuously weight seed aggregation and routing; the structure-gate floor is a training-only soft penalty.

Submitted closed-world Stage 2 objective and three-state evidence audit

The training objective follows the two-stage supervision design. The binary branch is trained on the full Stage 1 dataset, including curated positives and operational comparators. The target/effect-family branch is trained on the toxin-positive-peptide cohort with retained Stage 2 evidence. A sample-level mask indicates whether peptide i contributes to Stage 2 training.

The submitted checkpoints were trained on the original closed-world matrix: 3205 positive and 34 053 unretained cells entered binary cross-entropy (bce) and positive-versus-zero ranking. The final evidence audit did not retroactively alter these checkpoints. It reclassified 3126 cells as POSITIVE and 34 132 as UNKNOWN, with no VERIFIED_NEGATIVE cells, and defined audited evaluation subsets and revision-period sensitivity objectives. These objectives masked UNKNOWN cells or treated them as unlabeled, as specified for each loss (Supplementary Tables S34–S36; Supplementary Results S9). On the frozen 2292/287/287 split, we compared closed-world bce, positive-weighted bce, non-negative positive-unlabeled learning (nnPU), and unbiased positive-unlabeled learning (uPU); a three-epoch positive-only masked run remained diagnostic and was not tested.

The overall loss is

The submitted Stage 2 target/effect-family loss combines a sample-masked, closed-world multi-label sigmoid term, and a pairwise ranking term:

For the submitted checkpoint, the multi-label term is

The selected Full runs used independent sigmoid outputs rather than a 13-class softmax. Positive weights and square-root inverse-frequency class weights were estimated from the training split only and capped at 4.0; the five selected Full members used the same 13-label weight vector (Supplementary Table S25). For peptide i, let denote retained positive labels and denote unretained zero entries under the submitted objective. In submitted checkpoints, the observation mask o_ik was set to 1 for all 13 labels of every Stage 2-supervised peptide. The pairwise ranking term is

The selected Full runs used a ranking weight of 0.04 and margin of 0.2. Because the submitted objective treated unretained entries as zeros without verified broad-family negatives, probability, calibration, and closed-world discrimination metrics are interpreted as retained-versus-unannotated diagnostics. Hit rate at rank k (Hit@k) and mean reciprocal rank (MRR) are reported separately as retained-label recovery endpoints.

Confidence, auxiliary, and gate-regularization terms support training and operating diagnostics; they are not independent biological evidence.

Model selection and benchmark evaluation

Model families, features, hyperparameters, and operating thresholds were fixed on validation data before final test reporting. Stage 1 compared curated toxin-positive peptides with length-matched operational comparators. Area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC) were threshold independent; accuracy, balanced accuracy, F1, Matthews correlation coefficient, and confusion matrices used the frozen threshold.

Stage 2 evaluation was performed on the toxin-positive-peptide subset with retained target/effect-family evidence. For peptide i and label k, the model produced a score , and labels were ranked in decreasing order of . The primary endpoint was retained-label recovery in a short ranked list. Top-k success was defined as

where denotes the retained reference-label set for peptide i. MRR and normalized discounted cumulative gain@3 further characterized ranking. At fixed K = 1/3/5, retrospective yield was evaluated among peptides with at least one observed-positive family (Supplementary Tables S39 and S40); it does not measure realized laboratory cost.

Micro-AUPRC, micro-AUROC, and macro-F1 were secondary retained-versus-unannotated diagnostics. They assess coverage of curated reference labels, not biological absence or mechanism; thresholded Stage 2 rules were fixed before test reporting.

To examine cross-stage behavior without retuning either model, we applied the fixed Stage 1 threshold of 0.5 to the inherited 344-peptide submitted-label matrix from the strict Stage 2 test split. Oracle Hit@3 used all Stage 2-eligible peptides, whereas joint Hit@3 counted a peptide as successful only when it passed Stage 1 and Stage 2 ranked a retained label within the top three (Supplementary Table S33).

Recent implementations were retrained or adapted on the same MMSeqs90-controlled split when possible; released checkpoints were otherwise evaluated on that test split. ESM2, Evolutionary Scale Model Cambrian (ESMC), and the BERT-based pretrained protein language model ProtBERT were compared under the same frozen strict Stage 2 split, head recipe, and five seeds (Supplementary Tables S42 and S43; Supplementary Results S12). A strict-split 2 × 2 sensitivity crossed Protenix-v2 and AlphaFold3 graph sources at training and testing while retaining all five downstream seeds (Supplementary Table S48). These checks assess implementation sensitivity, not external validation or full reproduction of the original methods.

Stage 2 ablations used the frozen strict split and five fixed seeds. Input variants removed ESM2, MSA, structure/seed quality (with structure-derived scalars sanitized), or MSA plus structure/quality. Internal variants replaced MMoE with a parameter-matched shared MLP, label attention with fixed masked-mean readout, auxiliary evidential deep learning (EDL)/uncertainty/gate losses with none, or both MMoE and label attention. Other settings were fixed except inapplicable MMoE penalties. Primary analyses used the 340 final-audited supported peptides; the inherited 344-peptide matrix was secondary. Hit@1, Hit@3, and MRR were compared by paired bootstrap over the 58 strict-split assignment components (e1_component_id) shared across five seeds (10 000 replicates; seed 42), with separate Holm correction for the 12 input and 12 internal comparisons (Supplementary Table S49). EDL outputs from the no-auxiliary-loss variant were not interpreted. The prespecified resource panel comprised the first 32 target-supervised entries in the frozen strict Stage 2 validation manifest and was used only to measure deployment cost, not for model selection. It compared refit/reconstructed ESM2-LightGBM with a one-checkpoint Full path including database-MSA, five Protenix-v2 hypotheses per peptide, cache construction, and prediction serialization (Supplementary Tables S44–S46). The structure-quality and task-level MMoE fusion-allocation analyses are reported in Supplementary Table S45.

A source-restricted sensitivity analysis (Supplementary Tables S21–S23) preserved the MMSeqs90 split and comparators but trained and validated on one positive-source group before testing the other. These descriptive source/label-support stress tests are not independent-cohort evidence.

The external-evaluation protocol froze cohort rules, evidence criteria, model roles, checkpoints, and the Stage 1 threshold before inference. Sequence and similarity audits retained 87 external toxin-positive peptides (59 ConoServer; 28 ArachnoServer) and 549 operational comparators: 521 assay-scoped bioactive, 20 therapeutic, and eight disulfide-rich peptides. No endpoint-supported close non-toxic homolog met the formal criteria. Comparator roles are context-bounded rather than evidence of universal non-toxicity. The primary five-member Full ensemble was evaluated once at the fixed 0.5 threshold; cohort construction and sensitivity analyses are detailed in Supplementary Tables S26–S31. The submitted objective and source-separated external evaluation are documented in Supplementary Results S8.

Stage 2 external evaluation remained separate from Stage 1 and model selection. All 59 ConoServer and 28 ArachnoServer peptides were evaluated as separate cohorts with retained broad-family evidence, irrespective of Stage 1 passage. The preserved route used query-only multiple-sequence alignments in A3M format (A3Ms) without database homology search and therefore did not reproduce the database-MSA Full pipeline used for Stage 1. Hit@k and MRR measured retained-label ranking; micro-average precision remained a retained-versus-unannotated diagnostic. No external cascade score was computed without a complete cross-stage reference standard.

Probability calibration and selective-review diagnostics

Under the original submitted retained-versus-unannotated evaluator, temperature [28], Platt [29], and isotonic [30] scaling were fitted on validation data and applied unchanged to the test set; Brier score [31] and 15-bin expected calibration error (ECE15) summarized positive-versus-zero entries. Prespecified strict Stage 2 contrasts used paired bootstrap over the frozen grouping component (B = 10 000; seed 42), while objective-sensitivity and formal Stage 1/Stage 2 families used their specified paired resampling (B = 10 000; seed 20 260 805). Holm correction was applied within each family; fixed-0.5 McNemar tests remained separate. Family-level calibration under the final three-state policy required at least 20 positives and 20 verified negatives, excluding UNKNOWN; it was not estimated because no verified broad-family negatives were available (Supplementary Table S38).

For the original submission-split diagnostic [32], the prespecified five-member ToxiCompass-Full ensemble assigned each peptide a confidence score , where is the predicted score for label k. Test predictions were ranked by , and at each prespecified coverage c, the retained count was the nearest integer round(nc). Within each retained subset, top-k risk was defined as one minus the fraction of peptides for which at least one retained reference label appeared in the top-k predictions:

where is the retained subset at coverage c, and is the retained reference-label set for peptide i. The risk denominator included only retained peptides with an observed positive label; all-UNKNOWN peptides still counted toward coverage. Diagnostics were summarized by source and label family, with retained/deferred composition tracked (Supplementary Table S37). A separate strict-split robustness analysis ranked peptides by mean peptide-level binary-head EDL uncertainty for the corresponding ToxiCompass-Full ensemble; it was analysed separately. Deferred cases were assigned to human review, not interpreted as biologically safe.

Post hoc interpretability analysis

Post hoc interpretation of the validation-selected ToxiCompass-Full model used three analyses on the 287-peptide Stage 2 test set. Exact modality-level Shapley values enumerated all 64 presence/absence coalitions of the six native modality switches for retained true-label logits. Spearman correlations related the structure gate to mean pLDDT and inter-seed variance; 95% confidence intervals used 2000 peptide-level bootstrap resamples. TreeSHAP, a tree-specific implementation of SHAP (SHapley Additive exPlanations), provided an independent sequence-level view of the AAC/DPC LightGBM baseline. These analyses describe fitted-model behavior rather than causal mechanisms. Families with fewer than 20 observations were exploratory.

Results

Re-curated dataset and leakage controls

The benchmark contained 6114 Stage 1 samples: 3057 curated toxin-positive peptides and 3057 length-matched operational comparators. Of 2866 Stage 2 candidates, 2858 retained an observed-positive family label after audit, eight were all-UNKNOWN, and 191 positives were Stage 1-only. Exact-sequence merging and MMSeqs90 clustering yielded no exact overlap, shared cross-split clusters, or mixed toxin-positive/comparator clusters (Table 1).

Stage 2 labels were long-tailed and overlapping; five families met the main support threshold. Length matching reduced Stage 1 imbalance, although positive sources retained distinct cysteine and charge profiles (Fig. 2 and Supplementary Tables S19 and S20).

Figure 2.

Stage 2 label support and benchmark characteristics; (a) target/effect-family counts in the original closed-world evaluator (3205 positive assignments), shown separately from the final evidence audit reported in section Stage 2 target/effect-family ranking; (b–d) source-stratified length, cysteine-count, and net-charge distributions; (e) pairwise Jaccard overlap in the original evaluator; the benchmark remains strongly label-imbalanced and source-heterogeneous despite length matching.

Stage 1 binary toxicity screening

Stage 1 achieved AUROC/AUPRC 0.9871/0.9867 on the primary MMSeqs90-controlled test set (Table 2) and 0.9280/0.9469 on the family/accession/parent-analog sensitivity split; residual guard-qualified alignments preclude a fully guard-clean interpretation. On the inherited 344-peptide submitted-label matrix, the fixed 0.5 threshold retained 281 peptides and reduced oracle Hit@3 from 0.6860 to joint Hit@3 0.5669 (Supplementary Tables S32 and S33).

Table 2.

Stage 1 performance on the primary MMSeqs90-controlled test set (n = 612).

Model	AUROC	AUPRC	Balanced accuracy (BACC)	F1	Matthews correlation coefficient (MCC)
Label prior	0.5000	0.5000	0.5000	0.6667	0.0000
Length LightGBM	0.6241	0.6158	0.5000	0.6667	0.0000
Physicochemical LightGBM	0.9538	0.9460	0.8905	0.8878	0.7820
AAC/DPC LightGBM	0.9708	0.9580	0.9314	0.9318	0.8628
ESM2 LightGBM	0.9748	0.9725	0.9232	0.9223	0.8466
ESM2-k-nearest neighbors (KNN) k = 5	0.9614	0.9471	0.9069	0.9055	0.8141
ESM2-MLP replicate mean	0.9695	0.9703	0.9020	0.8993	0.8050
ToxiCompass Stage 1	0.9871	0.9867	0.9510	0.9515	0.9021

Threshold-dependent metrics use validation-frozen operating thresholds; controlled same-split implementations of published methods are reported in Supplementary Table S17.

Physicochemical and sequence baselines outperformed length alone (Table 2). On the strict split, Full exceeded the matched sequence-only control in AUROC (+0.0437) and AUPRC (+0.0457) after Holm correction, but fixed-threshold accuracy did not differ significantly (Supplementary Table S41). Published-method comparisons are in Supplementary Tables S14 and S17. The supporting internal benchmark results are presented in Supplementary Results S7.

Stage 2 target/effect-family ranking

Stage 2 included 2866 toxin-positive peptides (2292/287/287) and 13 families. The audit retained an observed-positive label for 2858 peptides and identified 3126 POSITIVE, 34 132 UNKNOWN, and no VERIFIED_NEGATIVE cells; family-level supported-positive test counts ranged from 1 to 90 (Supplementary Tables S34 and S36).

Under the original evaluator, ToxiCompass-Full achieved primary-split Hit@3/MRR of 0.8955/0.8011 (Table 3); final-label re-evaluation yielded 0.8920/0.7946. On the residual-overlap sensitivity split, the five-checkpoint mean-probability ensemble achieved Hit@3 0.6506 on 340 final-audited supported peptides. On the same cohort, fixed-budget retained-label recovery was 0.3765/0.6824/0.8118 at K = 1/3/5 (Supplementary Tables S39 and S40; Supplementary Results S11).

Table 3.

Stage 2 ranking and qualified secondary diagnostics on the primary MMSeqs90-controlled test set (n = 287).

Model	Hit@1	Hit@3	MRR	micro-AUPRC	macro-F1 (all labels)	Hit@3 at 80% coverage
Label prior	0.3171	0.6585	0.5276	0.2341	0.1188	0.7043
Length LightGBM	0.3763	0.6934	0.5761	0.3039	0.2115	0.7261
Physicochemical LightGBM	0.6272	0.8328	0.7506	0.6320	0.4154	0.8957
AAC/DPC LightGBM	0.6934	0.8815	0.8002	0.6977	0.4463	0.9391
ESM2 LightGBM	0.6760	0.9094	0.8020	0.6971	0.4814	0.9261
ESM2-KNN (k = 5)	0.7108	0.8850	0.8061	0.6732	0.4968	0.9217
ESM2-MLP replicate mean	0.6690	0.8955	0.7947	0.6747	0.5234	0.9348
ToxiCompass-Full	0.6829	0.8955	0.8011	0.7006	0.5583	0.9478
No-MSA/no-structure neural control	0.6376	0.9094	0.7795	0.6750	0.4609	0.9391

Note. Hit@k and MRR are retained-label ranking endpoints. Micro-AUPRC and macro-F1 are retained-versus-unannotated diagnostics under the original closed-world evaluator, while Hit@3 at 80% coverage is a selective-review diagnostic. Extended ranking and probability metrics are reported in Supplementary Table S24, and stratified coverage-risk results in Supplementary Table S37.

No revision-period objective variant significantly improved the primary ranking endpoints. The closed-world/no-ranking baseline gave five-seed Hit@1/Hit@3/MRR 0.6383/0.8857/0.7740; nnPU did not significantly change Hit@1 or MRR, whereas uPU reduced both after multiplicity correction (Supplementary Tables S34–S36). Full showed no Holm-adjusted strict-split advantage over no-structure or matched sequence-only controls (Supplementary Table S41).

Source-separated external evaluation

On the pooled external Stage 1 panel (87 toxin-positive peptides; 549 operational comparators), Full achieved AUROC/AUPRC 0.8977/0.7426 and, at the frozen 0.5 threshold, sensitivity/specificity 0.9540/0.4699 (Table 4).

Table 4.

Source-separated external Stage 1 toxin-like screening.

Toxin-positive n	Comparator n	AUROC	AUPRC	Sensitivity	Specificity	MCC
87	549	0.8977	0.7426	0.9540	0.4699	0.2960

Note. Results are from the five-member ToxiCompass-Full ensemble. Threshold-dependent metrics use the frozen 0.5 threshold. Confidence intervals and comparator-stratum results are reported in Supplementary Tables S26–S31.

With no complete cross-stage reference standard, external Stage 2 ranking was evaluated independently of Stage 1 passage. Hit@3 was 0.8136 for ConoServer and 0.7143 for ArachnoServer out-of-distribution (OOD) cohort (Table 5).

Table 5.

Source-separated external Stage 2 retained-label ranking.

Cohort	n	Hit@1	Hit@3	MRR
ConoServer	59	0.6271	0.8136	0.7432
ArachnoServer OOD	28	0.4643	0.7143	0.6168

Note. The cohorts were evaluated independently of Stage 1 passage. Confidence intervals and cohort-level audits are reported in Supplementary Tables S26–S31.

Representation and component robustness

ESMC gave the highest Hit@1 and Hit@3 point estimates under the controlled strict-split protocol (Table 6), but neither ESMC-ESM2 contrast remained significant after Holm correction. ProtBERT was lower than both backbones (all Holm-adjusted P < .0002; Supplementary Tables S42 and S43).

Table 6.

Controlled Stage 2 comparison of protein language-model backbones on the inherited submitted-label matrix (n = 344).

Backbone	Hit@1	Hit@3	MRR
ESMC	0.3924	0.7122	0.5794
ESM2	0.3605	0.6541	0.5500
ProtBERT	0.2355	0.4738	0.4204

Note. Values are five-seed mean-probability ensemble ranking results on the inherited matrix; analyses on the final-audited 340-peptide supported subset remain primary. Confidence intervals, paired contrasts, and embedding-resource measurements are reported in Supplementary Tables S42 and S43.

The 2 × 2 Protenix-v2/AlphaFold3 graph-source analysis showed no reliable source effect (Table 7). In the matched-source contrast (AlphaFold3 minus Protenix-v2), differences were −0.0233 for Hit@1 (95% CI –0.1620 to 0.0621), −0.0349 for Hit@3 (−0.1024 to 0.0370), and −0.0230 for MRR (−0.1018 to 0.0379). All intervals included zero, precluding superiority or equivalence claims.

Table 7.

Crossed structure-graph-source sensitivity on the inherited submitted-label matrix (n = 344).

Training graphs	Test graphs	Hit@1	Hit@3	MRR
Protenix-v2	Protenix-v2	0.3808	0.6860	0.5671
Protenix-v2	AlphaFold3	0.3663	0.6570	0.5526
AlphaFold3	Protenix-v2	0.3663	0.6512	0.5543
AlphaFold3	AlphaFold3	0.3576	0.6512	0.5441

Note. Values are mean-probability ensemble results on the inherited matrix and are secondary to analyses on the final-audited 340-peptide supported subset. Five-seed mean ± SD values, paired confidence intervals, and structure-consistency analyses are reported in Supplementary Table S48.

Among 340 final-audited supported peptides, only MSA removal changed a prespecified endpoint after multiplicity correction, reducing Hit@3 by 0.0282 (95% CI 0.0133–0.0551; Holm-adjusted P = .0024). No other ablation was significant (Fig. 3 and Supplementary Table S49; Supplementary Results S15). Results from the matched component-ablation analysis under the primary Stage 2 evaluation setting are provided in Supplementary Table S18.

Figure 3.

Paired Stage 2 ablations on the stricter sensitivity split; (a) input pathways; (b) internal components; effects are full minus ablation on 340 supported peptides, with 95% component-clustered bootstrap confidence intervals; Holm correction was applied separately within panels; the MSA-removal MRR interval excluded zero but was not significant after correction (P = .2222); only its hit@3 effect remained significant (P = .0024); complete results are in Supplementary Table S49.

Computational cost

On 32 peptides, the refit/reconstructed sequence reference required 23.19 s and the one-checkpoint Full path 651.64 s (28.10-fold) under the specified deployments. Five Protenix-v2 structure hypotheses per peptide accounted for 603.29 s (92.6%) of Full wall time (Supplementary Table S46). These structure-contribution and computational-cost analyses are detailed in Supplementary Results S13.

Calibration and selective-review diagnostics

Temperature scaling preserved ranking and changed retained-versus-unannotated Brier/ECE15 from 0.0513/0.0520 to 0.0512/0.0495; Platt and isotonic scaling reduced both but altered ranking. Final-policy family-level Brier/ECE10 were not estimable without verified broad-family negatives (Supplementary Table S38).

Under the original submitted-label evaluator, Hit@3 rose from 0.8955 at full coverage to 0.9649 at 20% (Fig. 4); final-audited values were 0.8920 and 0.9649. At 20%, retained cases were 87.7% BioTD-only and carried 58.3% Kv labels, with 3.5% UniProt-only. Strict-split Hit@1 risk was non-monotonic (0.6235/0.6488/0.6522 at 100%/50%/20%; Supplementary Table S37). Full audited curves and records are in Supplementary Fig. S2 and Supplementary Data S2; the selective-review stratification and calibration eligibility analyses are presented in Supplementary Results S10.

Figure 4.

Selective-review coverage-risk diagnostic for the prespecified five-member ToxiCompass-Full ensemble on the primary split using the original submitted-label evaluator; (a) retained hit@3; (b) mean confidence; (c and d) top-1/top-3 risk; dashed line: 80% coverage; final-audited curves, composition, and strict-split results are in Supplementary Fig. S2, Supplementary Table S37, and Supplementary Data S2.

Modality attribution and structure-quality associations

Exact Shapley attribution was broadest for ESM2 and second broadest for predicted structure (Fig. 5a and b). Kv/Nav labels were ESM2-dominant, enzyme/hemostasis labels showed greater MSA contribution, and the other-potassium-channel family showed the largest structure contribution. These values describe fitted-model information use, not incremental performance gains; the residual gate captures label-specific structural routing rather than all structural information propagated through fusion.

Figure 5.

Interpretability of ToxiCompass-Full (Stage 2 test set, n = 287); (a) exact modality-level Shapley contributions to true-label logits across supported target/effect-families (Kv, Kv channel; Nav, Nav channel; EH, enzyme/hemostasis; OK, other potassium channel; DM, membrane disruption); (b) per-family mean absolute modality Shapley values; (c) associations between the true-label structure gate and predicted-structure reliability (mean pLDDT; inter-seed structural variance); (d) family-level structure attribution versus median structure-gate use (bubble area, label support); analyses are post hoc attribution, not causal mechanism or experimental structural evidence.

The label-specific structural residual gate was small on the 287-peptide test set (median per-peptide mean true-label value 0.0017) but correlated with predicted-structure quality (Spearman r_s = +0.22 with mean pLDDT and −0.23 with inter-seed structural variance; both 95% CIs excluded zero; Fig. 5c and d). Exact TreeSHAP of the AAC/DPC LightGBM reference associated channel-toxin predictions with cysteine and disulfide-pair features and membrane-disruption predictions with cationic residues (Supplementary Fig. S1).

Discussion

ToxiCompass is a bounded framework for toxin-like screening and assay-family prioritization. Stage 1 screens against operational comparators, whereas Stage 2 ranks assay families among toxin-positive peptides. Lite and Full serve as reference implementations, and Stage 2 produces a shortlist for experimental follow-up.

Benchmark design supports this positioning. BioTD and reviewed UniProt records were re-audited, and MMSeqs90 cluster splitting controlled exact-sequence and cluster-level overlap under the specified protocol. Stage 1 AUROC/AUPRC decreased from 0.9871/0.9867 on the original split to 0.9280/0.9469 on the strict sensitivity split incorporating available source-accession, toxin-family, and explicit parent/parent-analog groups. An exhaustive post hoc audit nevertheless found 55 alignments meeting prespecified identity and coverage criteria across 19 evaluation peptides. The strict-split results are therefore a sensitivity analysis with residual sequence-overlap risk, not a fully overlap-free estimate (Supplementary Table S32 and Supplementary Data S1). In the source-separated panel, sensitivity was 0.9540 and specificity was 0.4699 at the frozen threshold. Interpretation remains limited by the small disulfide-rich stratum and the absence of an endpoint-supported close-homolog cohort. Stage 1 is therefore a sensitive first-pass filter whose performance depends on the comparator population, not a general safety classifier.

Stage 2 likewise supports a bounded ranking use case. Hit@3 decreased from 0.8955 on the original split to 0.6506 on the audited strict sensitivity split. The evidence audit identified 3126 supported-positive and 34 132 unknown peptide-label cells, but no verified broad-family negatives. Closed-world probability metrics thus measure retained-versus-unannotated separation rather than biological presence versus absence. Neither PU formulation improved the primary ranking endpoints: nnPU did not significantly change Hit@1 or MRR, whereas uPU reduced both after multiplicity correction. Hit@k and MRR remain primary because they test whether retained evidence is recovered within a short experimental shortlist without assuming exhaustive annotations.

Lite and Full address different deployment needs. In strict-split ablations, only MSA removal reduced Hit@3 after multiplicity correction; the data did not show reliable aggregate gains from ESM2, structure/quality, MMoE, label attention, or confidence-associated auxiliaries. Crossed AlphaFold3/Protenix-v2 results likewise showed no clear ranking benefit. On the monitored 32-peptide panel, the one-checkpoint Full path was 28.10-fold slower than the refit/reconstructed sequence reference under the specified deployments; generating five structure hypotheses per peptide accounted for 92.6% of Full wall time. Sequence-based inference is therefore the practical default, while Full provides optional structure-quality and multimodal context.

Several boundaries limit interpretation. Original-split selective-review gains coincided with BioTD and Kv enrichment, whereas strict-split EDL ordering did not yield stable risk reduction. The external cohorts provide retrospective robustness evidence rather than prospective blinded validation: Stage 1 retained high sensitivity but modest specificity, and Stage 2 ranking differed between ConoServer and ArachnoServer. A fully labeled external cascade is non-estimable because the comparator panel lacks Stage 2 family references. Audits identified non-zero source-release and inference-time database overlap that may have influenced representation quality (Supplementary Table S47). These exposures are distinct from supervised target-label leakage, but exact ESM2 sampled-training and Protenix-v2 optimization-step membership remain incompletely auditable. The results therefore support screening and assay prioritization within the evaluated populations, not universal safety assessment.

Future benchmark versions should address rare-family sparsity, the absence of verified broad-family negatives, and incomplete auditability of exact upstream training membership. Separately versioned releases should incorporate independent toxicity resources, endpoint-supported hard comparators, therapeutic peptides, and experimentally characterized designed sequences. Sequence-prediction, language-model, and multi-view methods [33–35] should be evaluated under the same evidence and split controls.

Conclusion

ToxiCompass links evidence-tracked toxin-like screening with bounded assay-family prioritization. Lite provides an efficient default, while Full offers optional structure-quality and multimodal context for case-level review. By making provenance, supervision, and label uncertainty explicit, the benchmark and its reference implementations provide a reproducible basis for comparing short-peptide screening and prioritization methods within defined evidence boundaries.

Key Points

• ToxiCompass provides an evidence-tracked two-stage benchmark for short-peptide toxicity screening and assay-family prioritization.

• Stage 1 screens against curated operational comparators under exact-sequence, length-matching, and Many-against-Many sequence searching 2 clustering at 90% sequence identity controls; Stage 2 produces bounded top-k shortlists of reference-supported assay families for toxin-positive peptides.

• On the original split, Stage 1 reached area under the receiver operating characteristic curve/area under the precision-recall curve 0.9871/0.9867 and Stage 2 Hit@3 0.8955; the residual-overlap strict-split sensitivity yielded Hit@3 0.6506.

• Lite supports rapid sequence-based screening and ranking, whereas Full adds multimodal context for diagnostic review rather than uniformly stronger prediction.

Abbreviations

AAC, amino acid composition; A3M, A3M multiple-sequence-alignment format; AMP, antimicrobial peptide; AUPRC, area under the precision-recall curve; AUROC, area under the receiver operating characteristic curve; BACC, balanced accuracy; BCE, binary cross-entropy; BioTD, online database of biotoxins; CI, confidence interval; DPC, dipeptide composition; ECE, expected calibration error; ECE10/ECE15, 10-bin/15-bin expected calibration error; EDL, evidential deep learning; ESM2, Evolutionary Scale Modeling 2; ESMC, Evolutionary Scale Model Cambrian; F1, F1 score; gPDE, global predicted distance error; GVP, geometric vector perceptron; Hit@k, hit rate at rank k; KNN, k-nearest neighbors; LightGBM, Light Gradient Boosting Machine; MCC, Matthews correlation coefficient; MLP, multilayer perceptron; MMSeqs2, Many-against-Many sequence searching 2; MMSeqs90, MMSeqs2 clustering at 90% sequence identity; MMoE, multi-gate mixture-of-experts; MRR, mean reciprocal rank; MSA, multiple sequence alignment; nDCG, normalized discounted cumulative gain; NLP, natural language processing; nnPU, non-negative positive-unlabeled learning; pLDDT, predicted local distance difference test; PLM, protein language model; PU, positive-unlabeled learning; top-k, the k highest-ranked predictions; uPU, unbiased positive-unlabeled learning; UniProt, Universal Protein Knowledgebase.

Author contributions

Qiule Yu (designed and implemented the ToxiCompass method; constructed the models and evaluated the performance; collected, compiled, and processed the data sets; analyzed the results; wrote the manuscript), Zhixing Zhang (analyzed the results; wrote the manuscript), Weihua Li (wrote the manuscript), Guixia Liu (wrote the manuscript), and Yun Tang (supervised the research project; wrote the manuscript)

Conflicts of interest

The authors declare that they have no conflicts of interest.

Funding

This work was supported by the National Natural Science Foundation of China [grant number U23A20530].

Code and data availability

Source code, pretrained models, the frozen benchmark, data splits, and documentation are available in the ToxiCompass GitHub repository (https://github.com/cihebiyql/ToxiCompass). The benchmark is archived as ToxiCompass Benchmark v1.0.0 (https://github.com/cihebiyql/ToxiCompass/releases/tag/benchmark-v1.0.0). A separate reproducibility-v1.1.0 release (https://github.com/cihebiyql/ToxiCompass/releases/tag/reproducibility-v1.1.0) contains the local ToxiCompass-Lite application and the resources required for inference. The public ToxiCompass-Lite web interface is available at https://lmmd.ecust.edu.cn/toxicompass/ and will be maintained for at least 3 years after publication. The versioned GitHub releases provide an offline alternative if the web service becomes unavailable.