VenusMutHub—A benchmark for protein mutation effect prediction

Protein engineering has become a cornerstone in numerous fields, from biocatalysis to biological drug development, offering innovative solutions with enhanced or novel protein functions¹. One of the critical challenges in this field is the prediction of protein mutation effects, a task of great importance for both drug development and precision medicine². The large mutational sequence space poses a significant challenge for traditional experimental approaches, underscoring the importance of computational strategies in protein engineering. Recent advancements in zero-shot computational methods, such as physics-based approaches and machine learning models, have substantially improved the prediction of protein mutation effects3, 4, 5. However, while these models perform well on large-scale mutation datasets, they face limitations when applied to real-world scenarios where high-throughput screening is not feasible or when specific biochemical properties need to be considered. The practical challenge lies in the small-scale, experimentally validated datasets often used in protein engineering, particularly in directed evolution efforts⁶. These datasets are more reflective of real-world constraints, where experimental data is sparse and derived from a limited set of mutations⁷. Thus, predicting mutation effects in these small-scale datasets, which more closely resemble the conditions protein engineers face, becomes a crucial task.

Recently, Zhang et al.⁸ present VenusMutHub, a benchmark study for protein mutation effect prediction. It represents a specific advancement in protein mutation prediction by providing a standardized, comprehensive framework for evaluating 23 computational models on 905 small-scale experimental datasets, which were curated from 527 unique proteins derived from published literature and public databases (Fig. 1). VenusMutHub covers four key functional properties: stability (59.7%), activity (19.3%), binding affinity (15.8%), and selectivity (5.2%). The evaluated models fall into three categories: sequence-only (e.g., ESM⁹), evolution-informed (e.g., GEMME¹⁰ and VESPA¹¹), and structure-aware (e.g., MIF¹² and VenusREM¹³). Performance was assessed using robust metrics, including Spearman correlation, normalized discounted cumulative gain, accuracy, and F1 score, to measure ranking, classification, and prediction consistency.

Figure 1.

The benchmark establishment and evaluation for predicting protein mutation effects.

They found that different models shine in different areas. For stability, structure-aware models (e.g., MIF) performed best, achieving high accuracy (0.627). For activity, evolution-informed models (e.g., VESPA) led with a strong correlation (0.338). Binding affinity predictions varied: multichain models excelled for protein–protein interactions, while various models demonstrated better average predictive capabilities for DTI (drug–target interaction) than PPI (protein–protein interaction). However, all models struggled with selectivity, showing very low correlations (0.099), due to the complexity of these predictions. The study also explored the impact of dataset size, finding that model performance improves significantly with datasets containing 8–13 mutations or more. Structure-aware models exhibited lower variance in stability predictions, making them more reliable for this property, while evolution-informed models were more consistent for activity predictions. These insights are critical for researchers selecting models for specific applications.

VenusMutHub is a transformative resource for protein engineering, offering a rigorous evaluation of computational models and practical guidance for their application. Its focus on small-scale, biochemically validated datasets bridges the gap between computational predictions and real-world needs, making it a valuable tool for biopharmaceutical development and precision medicine. The study's findings—that structure-aware models excel in stability predictions, evolution-informed models in activity, and multichain models in specific binding scenarios—provide actionable insights for optimizing protein design workflows.

However, the benchmark has limitations. The dataset's uneven distribution, with 59.7% of data related to stability and only 5.2% to selectivity, may limit its generalizability across all functional properties. Potential biases in the selection of protein families could also affect the applicability of the results. Moreover, the poor performance in selectivity predictions and the challenges in handling multiple mutations, which exhibit non-additive epistatic effects, highlight significant gaps in current modeling approaches. Future research should prioritize the development of hybrid models that integrate sequence, structure, and evolutionary data to improve prediction accuracy across all properties. Incorporating substrate-specific information through docking simulations could address the selectivity challenge, while uncertainty quantification would enhance model reliability by providing confidence measures alongside predictions. As the field progresses, VenusMutHub's open-access dataset will continue to drive innovation, encouraging the creation of next-generation models to tackle these persistent challenges and advance protein engineering.