Blind Challenges Let Us See the Path Forward for Predictive Models

Abstract

The rapid proliferation of AI/ML models in drug discovery heralds an era of extraordinary progress but also raises urgent questions about whether the true predictive performance is as good as advertised. On-target prediction models often benefit from high-resolution structural or atomistic representations that capture the subtleties of binding affinity and pose. In contrast, off-target and ADMET liabilities have typically relied on more implicit representations of molecular interactions. Retrospective benchmarks often provide a misleading picture of how successful these diverse representations are at predicting properties, and the community lacks standardized, prospective comparisons. Blind challenges, such as the OpenADMET × ASAP × PolarisHub Challenge featured in this issue, are crucial for realistically evaluating progress, encouraging iterations, and directing collective efforts toward major accuracy barriers. With ongoing investment in large-scale, open data creation, and community-led challenges, predictive modeling is poised to rapidly transform drug discovery by enabling accurate, multiparameter optimization.

Drug Discovery Is Costly

Drug discovery and development is costly, time-consuming, and subject to failure. While clinical phases are individually the most costly, the sequential nature of drug discovery and high cumulative failure rates mean that the majority of cost per approved drugwhere total costs per approved drug now exceed $1B , is due to accumulated failures in the discovery phase and associated issues that likely could have been addressed earlier in discovery. The success of a drug discovery program hinges on multiparameter optimization: the empirical balancing of on-target potency, off-target specificity, and ADMET properties. This balancing act is where predictive models have the greatest potential to reduce costly cycles of design, synthesis, and testing.

Computer-Aided Drug Discovery (CADD) Holds Enormous Potential to Accelerate Progress

Computer-aided drug discovery (CADD) has long sought to guide molecular design decisions with predictive models, aiming to save time and reduce attrition. Even modest improvements in model accuracy can yield superlinear returns by reducing the number of compounds that need to be synthesized or advanced. ,

A surge of enthusiasm for new drug discovery methods driven by artificial intelligence (AI) and machine learning (ML) has brought significant new talent, techniques, and energy to the field. This has increased expectations for major breakthroughs in performance, similar to what AlphaFold achieved with protein structure prediction (as well as the expansion into related areas, such as nucleic acid and small-molecule prediction, using software inspired by AlphaFold).

The history of blind challenges in related fields of structural biology and computational chemistry underscores their value. The CASP experiment in protein structure predictionsupported by decades of systematic data curation from the Protein Data Bank (PDB)galvanized progress and culminated in AlphaFold’s breakthrough. Blind challenges drive method development by providing essential experimental feedback, sharing valuable benchmarks with the community, and maintaining focus on aspirational goals for models that deliver real utility.

As CADD looks for its “AlphaFold” moment, it remains difficult to evaluate how well these models actually perform in practice. Retrospective benchmarks are plagued by data leakage, inconsistent curation, and a lack of standardized data sets. , Moreover, practical AI/ML models often demand large, high-quality data sets that remain scarce in many critical domains of drug discovery. Blind challenges provide an essential solution. By assessing models prospectively on common, well-designed data sets unavailable during training, blind challenges create a level playing field, generate realistic estimates of predictive utility, focus the field on critical problems in need of solutions, and foster rapid community-wide iteration and learning to accelerate progress. Similarly, SAMPL, D3R/CELPP, and CACHE have advanced free energy calculations, docking, and hit identification, respectively. Without careful prospective assessment, it is easy to fool ourselves into overestimating practical performancea risk Richard Feynman famously warned against (“The first principle is that you must not fool yourself and you are the easiest person to fool.”). Currently, the CACHE initiative has begun to fill this gap for on-target hit identification, providing a template for how prospective evaluation can sharpen models and align community efforts.

ADMET Properties and Antitarget as a Focus for Blind Challenges

While on-target binding is often the first focus of predictive modeling, the ultimate fate of a drug candidate is usually determined by ADMET properties and interactions with antitargets, proteins where drug binding alters toxicity and pharmacokinetics. Poor solubility, metabolic instability, transporter efflux, or unexpected channel binding (e.g., hERG) are among the leading causes of drug discovery failure. While factors such as poor solubility are primarily physicochemical, other critical bottlenecks, such as metabolic instability, transporter-mediated efflux, or unexpected ion channel binding, are often mediated by a relatively limited repertoire of proteins. While many of these liabilities are now screened for in the preclinical phase, they remain primary drivers of attrition in the discovery pipeline. We dub this specific set of proteins “antitargets”. This set includes targets that mediate toxicity and metabolizing enzymes (such as CYPs). While interactions with the latter are sometimes optimized for specific profiles, they represent a finite landscape of interactions that, if better predicted, would benefit the entire field. The limited size of this set suggests that understanding these liabilities could be essential to improving success rates. Incorporating ADMET and antitarget data into predictive frameworks ensures that the next generation of AI/ML models does not simply identify binders but also identifies compounds with a realistic chance of becoming safe and effective medicines. Moreover, any gains made in the ability to predict compound interactions with these antitarget proteins are likely to benefit all drug discovery efforts. In contrast, gains in the predictive ability against a specific target may not generalize.

Blind Challenges Require Evergreen Data Generation Efforts

For blind challenges to succeed in transforming the field, they require an evergreen source of new data. Retrospective repositories such as ChEMBL aggregate valuable information from the literature but often resemble “dumpster diving” for data: large numbers of small heterogeneous assay data sets, inconsistent conditions, correlated or biased data sets generated for an orthogonal purpose, and mixed measurement types complicate the ability to both build accurate models and assess predictive utility with appropriate statistical power.

Centralized, large-scale initiatives can overcome these limitations by generating large, robust, high-quality, consistent data sets tailored to predictive modeling needs. Economies of scale, advanced technologies, and active learning can reduce costs, increase scale, and ensure the data generated is highly informative and fit-for-purpose for building and assessing predictive models.

In the near term, we anticipate that individual academic groups will continue to generate valuable public data sets on specific targets, providing a critical testing ground for new models. However, the most pressing ADMET data sets remain locked behind closed doors in industry, limiting broad community impact. Initiatives like OpenADMET (currently funded by ARPA-H Avoid-ome, the Gates Foundation, and the Astera Institute), in partnership with ASAP (the AI-driven Structure-enabled Antiviral Platform) and PolarisHub, aim to break this barrier by generating open data sets that capture both structural and functional information on key antitargets. OpenADMET breaks down these barriers by leveraging high-throughput experimentation and structural biology to provide insights into how diverse small molecules bind to the “Avoid-ome”. By pairing X-ray and cryoEM structures with comprehensive biochemical and cellular assay data generated under consistent conditions, the initiative provides the community with the “ground truth” needed to move beyond reliance on heterogeneous literature databases. These collaborations create a continuous pipeline of high-throughput experimental data that can be used in future challenges and benchmarking efforts, ensuring that the challenges featured in this issue have a sustainable source of high-quality data. Sustained blind challenges on individual antitarget data sets will sharpen models for well-defined liabilities, while complementary ADMET challenges from individual target-based campaigns, such as the pan-coronavirus study highlighted here, will test whether models can handle the multiparameter trade-offs that ultimately determine success in drug development. Together, these dual-challenge formats will be essential to ensuring that predictive modeling keeps pace with the complex realities of drug discovery.

Research reported in this publication was partially supported by the Advanced Research Projects Agency for Health (ARPA-H) under AVOID-OME: Structurally enabling the “avoid-ome” to accelerate drug discovery and Award Number 1AY1AX000035-01.

The authors declare no competing financial interest.

Published as part of Journal of Chemical Information and Modeling special issue “Open Science and Blind Data: The Antiviral Discovery Challenge”.