Uncertainty Quantification in Molecular Machine Learning for Property Predictions under Data Shifts

Abstract

Drug discovery and medicinal chemistry efforts are increasingly influenced by machine learning (ML), with compound property prediction as a central application. ML models have demonstrated strong performance in predicting various compound properties from chemical structure. However, these models can exhibit varying levels of prediction error, making uncertainty quantification (UQ) essential for informed decisions. Standard UQ metrics include the distance to the molecules in the training set and prediction variance, obtained through methods such as model ensembles or Bayesian modeling. Although several UQ methodologies have been developed in recent years, no single approach consistently outperformed others. Herein, we present a comprehensive benchmark of UQ strategies for ML-based prediction of absorption, distribution, metabolism, and excretion (ADME) properties, using both in-house and public data sets. We employed the recently introduced UNIQUE (UNcertaInty QUantification bEnchmarking) framework and evaluated UQ method performance under data shifts. Our findings indicate data-based UQ metrics (e.g., chemical distance), and model-based UQ metrics (e.g., predicted value and variance) may capture complementary aspects of uncertainty. Their combination through error models, designed to predict the original ML model’s error, yielded higher-quality uncertainty estimates. These error models emerged as a promising strategy for enhancing UQ, showing robustness in under various degrees and types of data shift. Taken together, our work highlights the potential of combining diverse UQ metrics and error modeling to improve reliability in molecular property prediction. By establishing standardized evaluation setups and assessing UQ under data shifts, we provide a foundation for future UQ method development and benchmarking in the field.

1. Introduction

Machine learning (ML) models are routinely used for decision-making across disciplines, including drug discovery and medicinal chemistry. − In this context, compound property predictions from chemical structure are particularly relevant to guide molecule prioritization before synthesis and inform study selection. − Over time, ML models of varying complexity have been benchmarked and adopted in pharmaceutical research to identify promising drug-like molecules. − Despite strong performance across diverse properties and chemical series, ML models can still yield incorrect predictions. This highlights the importance of uncertainty quantification (UQ), which aims to assess the reliability of individual predictions on new compounds. − UQ has gained momentum with the rise of black-box ML and deep learning models, which may capture more complex structure–property relationships at the expense of interpretability. , Therefore, evaluating the reliability of their predictions becomes more challenging but increasingly relevant.

In cheminformatics, early efforts to identify unreliable predictions led to the definition of applicability domain metrics based on chemical distance to training set molecules. − This concept was inspired by the similarity-property principle, which states that “similar molecules have similar properties”. − These metrics relying on chemical space coverage by the training compounds are also termed data-based UQ metrics. Since distance refers to the metric computed in a given feature space, the choice of the molecular representation can significantly influence their interpretation and utility. Distances are typically defined in the input feature space of the models or in the learned model-internal representations. A recent study has shown that both are highly correlated, supporting the use of either for UQ. Even though a relationship between model performance and distance to the training set is often observed, distance does not generally provide accurate uncertainty estimates. , Models may fail on compounds close the training set due to undetected property cliffs, , while they may also extrapolate successfully to new (distant) chemical series that are far from the training molecules. This results to low distance but high error in some cases, and high distance with low error in others. When predictions’ reliability is assessed solely through chemical space coverage, compounds distant from the training set in the selected feature space are considered as out-of-domain or out-of-distribution (OOD). , Thus, those compounds are excluded from decision-making despite potentially representing the most valuable use cases for the model. Many data-based UQ metrics have been proposed based on distances to the training set, with Tanimoto similarity being among the most widely used. High correlation has also been observed across distance values calculated with different molecular representations and metrics. ,, Other data-based UQ strategies include kernel density estimation ,, or clustering instead of explicit distance values.

Model-based UQ metrics, which derive uncertainty from model outputs, have also become popular. These encompass multiple ML methods that output uncertainty estimates, namely Bayesian models − or ensemble models such as random forest, deep ensembles, ,− and Monte Carlo dropout. , Gaussian processes have also been widely used for UQ assessments, − and the popularity of graph neural network (GNN) ensembles and mean variance estimation has also increased in recent years. All these methods report a predicted value along with a full distribution or a variance as a proxy for uncertainty. However, previous studies have shown a lack of correlation between models’ prediction variance and true error, which may partly result from the assumption of normal data distributions underlying many UQ metrics. , Moreover, model variance typically ignores the bias component of the uncertainty arising from model misspecification rather than parameter uncertainty. ,,

To potentially overcome this limitation, error models (EMs) have also been investigated as uncertainty estimators, but conclusions have differed between studies. − EMs are ML models trained to predict the error of the original model’s predictions and have been mostly built with the same molecular representation as the original ML model. − Overall, hybrid UQ approaches that combine training data coverage and model-derived information remain relatively unexplored.

Numerous benchmarking studies have shown that no single method consistently outperforms others across data sets and endpoints. ,, Roth and Bajorath recently emphasized the need of future research on advanced UQ metrics, including alternative evaluation strategies and test systems. The UNIQUE (UNcertaInty Quantification bEnchmarking) framework was introduced to address this gap by facilitating the combination of data- and model-based UQ metrics, and standardizing UQ benchmarking. UNIQUE is an open-source Python library that supports UQ evaluations tailored to different application needs.

A critical yet often overlooked aspect of UQ is its robustness under data shifts, which refer to systematic differences in data distributions between training and test sets. These shifts can arise from changes in chemical space or property ranges and often occur when models are applied to new compound series. They are commonly encountered in compound property data relevant to pharmaceutical research, − where models trained on historical data must generalize to novel chemical scaffolds or assay conditions. Data shifts challenge the assumption that training and test data are drawn from the same distribution, making UQ especially critical under these conditions. Friesacher et al. recently showed that UQ methods performing well in-distribution (ID) may not generalize to OOD compounds. Nevertheless, previous research has primarily emphasized calibration rather than the ability of UQ metrics to rank predictions by model error, with an emphasis on classification tasks. ,, While UQ metrics can be recalibrated, ,, their utility is limited if they are not indicators of model error. , Hence, there is still a need for benchmarks that reflect prospective use cases and data shifts in molecular ML for property predictions.

In this work, a large-scale benchmark of UQ metrics is presented for ML-based property prediction, using both in-house and public data sets. Standard UQ metrics are compared to hybrid strategies that merge UQ information from the training data coverage and model outputs. UQ method evaluation focuses on their ability to inform about ML predictions’ error, which is critical for decision-marking. Recommendations are provided for evaluating UQ methods under data shifts of different nature, supported by analyses on realistic temporal validations with large Novartis’ ADME data sets.

2. Materials and Methods

2.1. Data Description

2.1.1. In-House Data Sets

Experimental data were assembled for seven physicochemical and ADME-related properties measured through various in vitro Novartis’ assays. The endpoints considered were intrinsic clearance in rat and human liver microsomes (rLM and hLM CL_int), rat and human plasma protein binding (PPB), apparent permeability from the low-efflux Madin–Darby canine kidney cells (MDCK) assay (LE-MDCK P_app), efflux ratio in MDCK-MDR1 cells (MDR1 ER), and octanol–water partition coefficient (LogP). When multiple measurements were available for the same compound and endpoint, the geometric mean was considered. Values outside the dynamic range of the assays were excluded for CL_int and PPB assays, and qualified values (“<”/“>”) were discarded for LE-MDCK P _app and PPB. Fraction unbound (f _u) in plasma was calculated from PPB values, and logarithmic transformations were applied to all assay endpoints, except to LogP. Additional basic preprocessing steps were applied, including the standardization of Simplified Molecular Input Line Entry System (SMILES) and removal of salts. The final data set under analysis comprised 293,569 unique compounds.

2.1.2. Public Data Sets

Open-source data for five ADME endpoints, namely rLM CL_int, hLM CL_int, human and rat PPB, and MDR1 ER, were retrieved from Fang et al. PPB values were also transformed to f _u values and logarithmic transformations were applied to all end points. The open-source data set used for our analyses contained 3517 unique compounds.

2.2. Data Splitting

For both in-house and open-source data sets, compounds were divided into three subsets: training (model building), calibration (uncertainty-related calibrations), and test (evaluation) sets. For a realistic assessment of model predictivity and UQ performance, a temporal split was applied for in-house data , according to compounds’ registration date in the Novartis’ database. Training and calibration sets were comprised by molecules registered and measured until the end of 2022 and 2023, respectively. Test sets consisted of molecules registered from and measured between January 2024 until April 2025. Figure S1 reports the data subset distributions in feature and property space. While property distributions remain relatively stable across subsets, compounds become progressively more diverse compared to the training set. This trend is reflected in the increased distance of calibration and test sets to their five nearest neighbors (5-NN) in the training set. As further detailed below, distances to the 5-NN were calculated based on the Manhattan metric and model-internal representations. These observations highlight the importance of assessing UQ quality under these realistic data shifts. For public data sets, a scaffold-based approach was applied, grouping compounds with the same chemical scaffold into the same subset. This data splitting also challenges the ML models and UQ methods, which are evaluated to predict and estimate uncertainty for new molecular scaffolds. Public training, calibration, and test sets contained 50%, 30%, and 20% of the compounds, respectively. Table reports the number of molecules per each data source, subset and endpoint.

1. Data Descriptive Statistics .

endpoints	data source	# training set compounds	# calibration set compounds	# test set compounds
rLM CLint	in-house	199,981	12,684	20,590
	public	1527	915	609
hLM CLint	in-house	137,029	12,755	19,967
	public	1544	925	616
rat f u	in-house	12,385	1160	1402
	public	87	48	32
human f u	in-house	9476	1008	666
	public	96	58	39
LE-MDCK P app	in-house	34,352	14,439	21,597
MDR1 ER	in-house	415	1182	1559
	public	1321	925	527
LogP	in-house	34,952	5006	7550

^aReported are the number of unique compounds per each endpoint under evaluation, data source and subset (training, calibration, and test sets).

2.3. Machine Learning Models

2.3.1. Model Building

Graph neural network (GNN) regression models were developed for the prediction of physicochemical and ADME properties. ,, The models consisted of a directed message passing neural network followed by a feed-forward deep neural network, built with the chemprop package (version v2.1.2) and the PyTorch framework. With this methodology, property predictions are only based on compound structure ^8,12,17and, after a message-passing phase, a learned representation is derived from molecular graphs. In all models, the latent GNN representation consisted of a 300-dimensional vector, which was input to the feed-forward neural network (readout phase) to generate predictions. All models were GNN ensembles trained with different random initializations and early stopping.

Multitask learning enables the modeling of multiple properties simultaneously. ,, Here, four multitask GNN models were built for the in-house data: Clearance (6-task model; rLM and hLM CL_int under analysis), Binding (10-task model; rat and human f _u under analysis), Permeability (6-task model; LE-MDCK P _app and MDR1 ER under analysis), Lipophilicity (2-task model; LogP under analysis). Additional model details are provided in Supporting Information, including the models’ architecture and hyperparameters (Table S1). These models correspond to our Novartis’ global models, used internally to predict compound properties. Before internal model deployment, various strategies were benchmarked. The development and evaluation of these models have been discussed in previous publications. ,,, For open-source data sets, GNN regression models were built for CL_int (2-task model with rLM and hLM CL_int), Binding (2-task model with rat and human f _u), and MDR1 ER (1-task model). Default hyperparameters from the chemprop library were used.

2.3.2. Model Evaluation

ML models’ errors on property predictions were prospectively evaluated on the test set using the mean absolute error (MAE).

$$\mathrm{M}\mathrm{A}\mathrm{E}=\frac{1}{n}\sum _{i=1}^n|y_i-{ŷ}_i|$$ 1

where y is the experimental value, $ŷ$ is the predicted value, and n is the number of test compounds. Geometric mean fold errors (GMFE), equivalent to the MAE in linear scale (GMFE = 10^MAE) and more common in ADME-related literature, were also calculated. Predictions within 2-fold of the experimental value are typically considered within experimental variability. As a control, predictive performance was also assessed for a baseline model that consistently outputs the average property value from the training set for any given input compound (mean baseline).

2.4. Uncertainty Quantification

2.4.1. UQ Metrics

UQ metrics of different nature were calculated and evaluated on the test set. Two standard UQ metrics (data-based, model-based) were considered.

• Manhattan distance to the 5-NN in the training set based on the GNN model’s latent representation (data-based UQ metric).

• Ensemble variance from the GNN model (model-based UQ metric).

Hybrid UQ metrics, including error models, were also designed to explicitly combine data- and model-based UQ information.

• Sum of variances consists in the summation of the ensemble variance and the Manhattan distance to the 5-NN in the training set (transformed to a variance). The distance was converted to variance using the calibrated negative log-likelihood on the calibration set, as reported in Hirschfeld et al. and implemented in UNIQUE. , Both individual components were normalized using Robust Scaler from scikit-learn prior to their aggregation.

• DiffkNN was calculated as the absolute difference between the predicted value (or variance) for a test compound and the predicted value (or variance) of the k-NN in the training set. These two versions of DiffkNN with predicted mean or variance from the model ensemble were calculated with k = 5 and were referred to as Diff5NN-Pred and Diff5NN-Var, respectively. , The choice of 5-NN for distance-based metrics was further retained for consistency across benchmarking scenarios, following its original definition in prior work by Sheridan, , where Diff5NN demonstrated strong correlation with model error.

• Error models (EMs): random forest (RF) models were generated to predict the original model’s absolute error. The absolute prediction errors for the property model on the training and calibration sets were used as labels. EMs were built with the following input features: (i) Manhattan distance to the training set, (ii) ensemble variance, (iii) predicted value from the original GNN model. Thus, EMs are trained in a supervised fashion to learn optimal weights for combining these UQ metrics. The scikit-learn RF implementation was used, with 50 trees and a maximum depth of 10.

2.4.2. UQ Evaluation

ML model’s true error was considered as ground truth for our uncertainty assessments, and UQ metrics were evaluated on their ability to distinguish between predictions of varying error levels, also known as ranking-based evaluation. , Spearman’s rank correlation coefficient (SRCC) estimates the ability of each UQ method to rank predictions according to the original ML model’s errors.

Bootstrap samples (n = 100) were generated on the test set, obtaining a distribution of SRCC values per each UQ metric. Pairwise comparisons of these distributions were performed using two-sided Wilcoxon rank-sum tests, with Benjamini-Hochberg correction to control for multiple testing. The best UQ metric was defined as the one that most frequently showed significant superiority in pairwise comparisons. , Although ties were possible, in practice a single UQ metric always outperformed the rest. Beyond statistical significance, differences were also assessed for practical relevance. SRCC values are bounded and reflect correlation strength, but whether differences in their means are practically relevant is less obvious. Thus, the effect size was calculated through the Cohen’s D coefficient between each pair of UQ metrics. Cohen’s D is a unitless measure that standardizes the difference in means by considering the variance of both distributions, and values of 0.2, 0.5, and 0.8 or greater indicate small, medium, and large effect sizes, respectively.

To further assess the utility of UQ metrics, a null UQ method was constructed by applying a permutation test to the model’s absolute true errors. The errors were shuffled and SRCC values were computed between the original and permuted errors to generate a baseline SRCC distribution (permutations’ test baseline). Comparing the SRCCs of actual UQ metrics against this baseline allows us to determine whether they provide information beyond random noise.

Additional ranking-based evaluation metrics such as the performance drop and area under the curve difference were also calculated. ,,

2.4.3. UNIQUE Setup

The UNIQUE framework for UQ benchmarking was used for the calculation of various UQ metrics as well as evaluation analyses (Python package v0.2.2). UNIQUE’s input file contained the following information (columns): (i) compound identifiers (IDs), (ii) property values (labels), (iii) model predictions, (iv) the subset memberships used during model building (training, calibration or test), (v) the scaled GNN latent fingerprints (feature input type), and (vi) prediction ensemble variance (model input type). UNIQUE provided a collection of UQ metrics, as well as summary tables and plots for evaluation.

2.5. Data Shifts

Apart from evaluations on the complete test sets, UQ methods were also evaluated under increasing feature, label, discontinuity, and error shifts. Compounds were sorted by a specific shift criterion and SRCC values were calculated at increasing percentages of test data. Data shifts were defined as follows.

• Feature shift: Manhattan distance to the 5-NN in the training set based on the GNN latent representation.

• Label shift: difference between a compound’s property value and the training set’s central tendency, defined as the median of the property values (labels).

• Discontinuity shift: compounds were sorted according to the Diff5NN metric applied to the labels, which was introduced by Sheridan et al. to identify “activity cliffs”. This metric quantifies the absolute difference between the measured property of each test compound and the mean property value of its 5-NN in the training set.

• Error shift: compounds were sorted according to the absolute true model error.

Figure schematizes the UQ benchmarking protocol applied in our work, including data set preparation, GNN model building and evaluation, and UNIQUE pipeline execution for the calculation of various UQ metrics, namely distance to the training set, ensemble variance and error models. Finally, UQ methods were compared on their ability to rank predictions according to model error (SRCC) on independent test sets and under specific distributional shifts.

1.

Calculation protocol for UQ methods’ benchmarking. For each data set, GNN models were trained and evaluated. The UNIQUE library for UQ benchmarking was applied to calculate multiple UQ metrics (data-based, model-based, and error models) and obtain evaluation scores, including SRCC. Correlations between ML model’s errors and uncertainty estimates were evaluated on independent tests sets as well as under data shifts (feature, label, discontinuity, and error). This workflow was applied both to in-house and public ADME data sets.

3. Results and Discussion

ML models based on graph neural network (GNN) ensembles were developed for the prediction of various ADME endpoints. Predictive performance and uncertainty quantification (UQ) evaluation analyses were carried out with in-house Novartis’ and open-source data sets, including seven and five properties, respectively, For Novartis’ models, a temporal data splitting strategy was implemented, where evaluations were done with the most recent compounds. For open-source data sets, a scaffold-based splitting approach was considered.

3.1. Models’ Predictive Performance

Table shows the models’ prediction errors on the test set in terms of the mean absolute error (MAE), its linear variant geometric mean fold error (GMFE), and the percentage of predictions within 2-fold of the experimental value (% 2-fold). Results are also reported for a mean baseline consisting of a constant prediction value of the mean property value in the training set.

2. Models’ Predictive Performance on the Test Set .

endpoint	data source	GMFE	MAE	MAE (baseline)	% 2-fold	% 2-fold (baseline)
rLM CLint	in-house	2.16	0.33	0.57	54.04	25.11
	public	2.69	0.43	0.63	41.71	26.60
hLM CLint	in-house	2.10	0.32	0.53	55.11	26.64
	public	2.33	0.37	0.53	49.51	27.11
rat f u	in-house	2.23	0.35	0.53	53.35	32.74
	public	3.54	0.55	0.54	40.63	40.63
human f u	in-house	2.27	0.36	0.56	53.75	27.48
	public	4.02	0.60	0.59	33.33	35.90
LE-MDCK P app	in-house	1.79	0.25	0.55	70.13	21.58
MDR1 ER	in-house	1.89	0.28	0.38	64.72	48.11
	public	2.41	0.38	0.58	48.20	24.29
LogP	in-house	2.33	0.37	0.95	54.57	19.40

^aPerformance values are reported for the ML property models built with in-house data (seven endpoints) and public data (five endpoints). Mean absolute error (MAE) is reported together with its linear variant, the geometric mean fold error (GMFE). The percentage of predictions whose predicted/experimental ratio lies within twofold (1/2 < x < 2) of the experimental values (% 2-fold) is also shown. The MAE and % 2-fold metrics are also reported for a control baseline, which consistently predicts the mean property value in the training set.

In-house models produced overall accurate predictions with GMFE ∼2-fold in most of the cases, which typically resembles experimental variability. Moreover, MAE and % 2-fold for ML models were consistently superior compared to the control baseline. Specifically, MAE values for baseline predictions ranged from 0.38 (MDR1 ER) to 0.95 (LogP). In contrast, the largest prediction errors for ML models were observed for LogP (MAE = 0.37), which also corresponded to the assay with the widest dynamic range. The largest differences between average ML errors and baseline predictor errors were observed for LogP (ΔMAE = 0.58) and LE-MDCK P _app (ΔMAE = 0.30). In contrast, ML-based predictions for MDR1 ER were closer to the control baseline (ΔMAE = 0.10). Notably, this endpoint had the smallest training set, which may have contributed to the observed performance.

In the case of publicly available data, ML models had MAE values ranging from 0.37 (hLM CL_int) to 0.60 (human f_u) and improved baseline predictions for three endpoints, namely rLM CL_int, hLM CL_int and MDR1 ER. For the f _u endpoints, ML-based predictions were associated with average errors between 3.5- and 4-fold, leading to similar errors as the mean baseline predictor (ΔMAE = −0.01). Models built with ADME public data sets consistently showed lower performance compared to their proprietary counterparts, despite the more realistic prospective validation setup.

Overall, ML models produced accurate predictions across various ADME properties, with superior performance to random guessing for all endpoints except f_u predictions on public data. These ML models, built on diverse data sources and endpoints, exhibit varying levels of predictive performance. This makes the ML setup well-suited for benchmarking UQ methods across ADME models of differing quality.

3.2. Uncertainty Evaluation

Uncertainty estimates were obtained through various methods and evaluated on their ability to rank test compounds according to prediction error. Specifically, absolute errors from the GNN property predictions served as ground truth. Ideally, low and high uncertainty estimates should be obtained for compounds associated with low and high model errors, respectively. The behavior of relative ranking according to model’s prediction error, which is key in decision-making applications, was evaluated with the Spearman’s rank correlation coefficient (SRCC). Figure shows the SRCC values on the test set for both in-house and public data sets, using bootstrapping to assess statistical significance across UQ methods. This figure focuses on the most promising UQ metrics, including distance to the training set, ensemble variance, and error models (EMs), as further discussed below. Additional results are reported in Figure S2, where pairwise SRCC values are shown across all evaluated UQ metrics as well as absolute model errors.

2.

Evaluation and comparison of uncertainty quantification metrics. Spearman’s rank correlation coefficient (SRCC) values assessing the correlation between true absolute model errors and uncertainty estimates are reported for the: distance to the training set (orange), ensemble variance (green), and error model-based estimates (EM, red). The evaluation of uncertainty estimates is reported for both (A) in-house and (B) public data sets. A distribution of SRCC scores was obtained for each UQ metric by performing bootstrap resampling on the test set (100 bootstrap samples). Statistically significant top-performing UQ methods were identified by conducting Wilcoxon ranked sum tests with subsequent Benjamini-Hochberg correction for multiple testing. Each boxplot displays the median, upper and lower quartiles, and whiskers extending to 1.5 times the interquartile range. Baseline performance (blue) is calculated using permutation testing on absolute model error.

3.2.1. Standard UQ Metrics: Distance and Ensemble Variance

The distance to the training set and variance of ensemble models are two of the most common UQ metrics. Distance can be defined in various feature spaces, including model-internal latent representations. In our study, the latent representation from the GNN model was consistently used, and the Manhattan distance to the five nearest neighbors (5-NN) was calculated in such feature space. Moreover, the prediction variance from GNN ensemble models was considered. Both the distance and the ensemble variance were evaluated and low correlations with model error were observed. For the distance to the training set SRCC values ranged from −0.08 to 0.20, whereas for ensemble variance SRCC values ranged from −0.07 to 0.26 across the different endpoints and data sources (Figure S2). Ensemble variance outperformed distance to the training set in some properties, namely rLM CL_int, hLM CL_int both for proprietary and public data sets, or LE-MDCK and MDR1 ER for in-house data (Figure ). Overall, although standard UQ metrics might offer some insights into model error for certain endpoints, our results show that they are often insufficient to provide accurate estimates of predictive ability, as also reported in previous studies. ,,,

A weak correlation was also observed between distance to the training set and model variance themselves, with SRCC values between 0.14 and 0.38 (Figure S2). This may suggest that these two UQ metrics may capture complementary aspects of uncertainty. To investigate whether their combination could improve uncertainty estimation, hybrid UQ metrics and EMs were explored.

3.2.2. Hybrid UQ Metrics: Sum of Variances and Diff5NN

Hybrid UQ metrics provided by the UNIQUE framework were computed to explicitly combine uncertainty information related to both training data coverage and model outputs: sum of variances and Diff5NN. The sum of variances was calculated by combining the distance (converted to a variance) and the ensemble variance. Additionally, two Diff5NN variants were obtained by calculating the absolute difference in variance (Diff5NN-Var) and predictions (Diff5NN-Pred) between a test compound and its 5-NN in the training set. Despite the rationale behind combining complementary sources of uncertainty, SRCC values for these hybrid UQ metrics were generally low across endpoints (Figure S2), indicating limited or no improvement over standard UQ metrics.

3.2.3. Error Models

EMs were investigated as an alternative approach to integrate information from the training data coverage and model outputs. , More specifically, random forest models were built to predict the absolute true error based on three input features: the distance to the training set, the predicted value (ensemble mean) and the ensemble variance. Figure shows that EMs consistently outperformed common UQ metrics for six out of seven endpoints for in-house data and across all cases for open-source data. SRCC values ranged from 0.06 (Rat f _u) to 0.39 (LE-MDCK P _app) for proprietary data, and from 0.16 (hLM CL_int) to 0.46 (MDR1 ER) for public data. While open-source data sets were smaller (Table ), which sometimes resulted in predictive performance near the mean baseline (Table ), EMs’ uncertainty estimates still identified predictions associated with higher or lower errors. Hence, EMs remained effective in estimating uncertainty across data sets of varying sizes, including low-data scenarios.

EMs showed both statistical and practical significance compared to standard UQ metrics, with corrected p-values below 0.05 across all endpoints and large observed effect sizes (Figure S3). Alternative evaluation metrics were also calculated, namely the area under the curve difference (Table S2) and performance drop (Table S3), which also identified EMs as the top-performing UQ strategy across most endpoints.

Our findings show that EMs were able to learn optimal combinations of various uncertainty indicators, which led to improved performance. Notably, if the underlying UQ metrics were redundant, their combination would not yield better uncertainty estimates, regardless of the training procedure. Therefore, the advantage of EMs reflects the complementarity between diverse sources of uncertainty.

3.2.4. Permutations’ Test Baseline

Low to moderate correlation values may stem from Normal distribution assumptions and the fact that high uncertainty does not always imply high error. , A permutations’ test baseline was introduced to contextualize SRCC values and determine which SRCC ranges informative of model error (i.e., better than random guessing). Permutations were applied to the true absolute errors from the original ML models, and SRCC values were computed between the true errors and the shuffled (permuted) versions. Interestingly, not all UQ metrics demonstrated higher performance than such permutations’ test baselines. These findings highlight the importance of evaluating standard UQ metrics before their usage, since they might be uninformative of the ML’s model error. Importantly, in all cases under consideration, EMs’ performance was superior to the permutation test baseline. Hence, in our benchmark analyses EMs always provided informative uncertainty estimates.

3.3. Assessing the Utility of Uncertainty Estimates

Despite EMs’ superiority over alternative metrics, the magnitude of SRCC values highlight the intrinsic complexity of developing an accurate UQ approach to rank predictions according to model error. To generate further intuition about whether EMs are useful in practice, the relationship between model error and uncertainty estimates was characterized. Figure shows the MAE values at increasing percentages of test compounds. Data points were added either from high to low uncertainty (left) or from low to high uncertainty (right), based on the EMs’ uncertainty estimates. MAE values consistently decreased when compounds with higher uncertainty were added first. Thus, EMs effectively identified high-uncertainty predictions associated with larger errors. Conversely, MAE values increased when test compounds were added from low to high uncertainty, further highlighting EMs’ ability to capture the reliability of low-uncertainty predictions, which were associated with lower errors.

3.

Relationship between uncertainty and model error. Results are shown for rLM CL_int (blue), hLM CL_int (orange), LE-MDCK P _app (green), and LogP (red) for (A) in-house data sets, and for rLM CL_int (blue), hLM CL_int (orange), and MDR1 ER (yellow) for (B) public data sets. MAE is shown in the y-axis at increasing number of compounds in the test set (% test compounds, x-axis). Compounds are added to the test set from high to low EM estimated uncertainty (left) or from low to high EM uncertainty (right), and cumulative model errors are reported.

Table S4 reports the MAE values for the highest and lowest uncertainty predictions considering three equally sized bins (33% each) based on the EMs’ uncertainty estimates. Differences in MAE scores between low- and high-uncertainty bins reached up to 0.26 for the in-house LogP data. Notably, both rat and human f_u public models, which demonstrated the lowest predictive performance across endpoints (Table ), resulted in two of the largest differences in average ML errors between bins (ΔMAE = 0.25 and 0.33, respectively). These findings further highlight the practical value of EM uncertainty estimates in distinguishing among compounds with varying levels of predicted errors regardless of the overall accuracy of the original ML model, which could be useful for compound prioritization even in property models with only moderate accuracy.

3.4. Prediction Uncertainty under Data Shifts

Data shifts pose a major challenge for ML, making reliable uncertainty estimates especially valuable. In previous sections, UQ metrics were evaluated under certain shifts, such as temporal splits in the in-house data sets. Although this already represents a realistic distributional change commonly observed in pharmaceutical applications (Figure S1), a protocol to benchmark UQ metrics under specific types of data shifts was implemented. Four types of data shifts were evaluated: (i) feature shifts, based on compound distance to the training set; (ii) label shifts, using deviation from the training set’s median; (iii) discontinuity shifts, identified via the Diff5NN metric based on the labels to capture structure–property cliffs, and (iv) error shifts, assessing UQ performance at increasing true model errors. These scenarios enable benchmarking of UQ methods under diverse shifts and challenging conditions potentially encountered in future model usage.

Data shift analyses were conducted on the largest four in-house data sets (rLM CL_int, hLM CL_int, LE-MDCK P _app and LogP). Figure S4 shows the distribution of the different types of shifts in the test sets. They were generally comparable across shift types and endpoints, with the largest differences arising from variations in the property dynamic ranges. Outliers were observed in all distributions, indicating the presence of out-of-distribution (OOD) compounds compared to the rest of the test set. Rather than applying a binary in-distribution (ID) or OOD definition, the performance of UQ methods was analyzed along a gradient of distributional shifts. In this setup, the most shifted compounds represent the tail of the distribution and effectively serve as OOD examples, allowing UQ performance to be evaluated under increasingly challenging conditions. Rank correlation between uncertainty estimates and true model error (SRCC) was assessed for increasing percentages of test compounds sorted by a specific shift criterion.

Figure reports SRCC values at different levels of data shifts for rLM CL_int and hLM CL_int endpoints. Results for LE-MDCK P _app and LogP are shown in Figure S5. EMs consistently outperformed distance and ensemble variance metrics across endpoints, demonstrating greater robustness to distributional changes regardless of the data shift type and magnitude.

4.

. Uncertainty evaluation under data shifts. Results are shown for (a) rLM CL_int and (b) hLM CL_int endpoints. Spearman’s rank correlation coefficients (SRCC) quantify the correlation between true absolute model errors and uncertainty estimates for three UQ metrics: distance to the training set (orange), ensemble variance (green), and error model-based estimates (EM, red). The y-axis displays SRCC values calculated at increasing percentages of compounds in the test set (x-axis). Compounds are added to the test set from higher to lower feature, label, discontinuity, and error shifts, and SRCC values are reported every 1%.

3.4.1. Feature Shifts

EM uncertainty estimates maintained stable correlations with true error across the entire range of chemical distances, even for the most shifted or OOD compounds. Only in the case of rLM CL_int, EMs showed a performance drop at extreme feature shifts (Figure a), while hLM CL_int (Figure b), LE-MDCK P _app (Figure S5a), and LogP (Figure S5b) remained unaffected. This represents a meaningful difference compared to other UQ metrics, which showed a drop in performance for the most shifted samples. This behavior was also reported by Tossou et al., where ensemble variance failed to accurately estimate the reliability for OOD compounds.

3.4.2. Label Shifts

Although EMs consistently ranked as the best-performing UQ metric across data sets, endpoint-specific variations highlight different relationships between property values and model errors over assays’ dynamic range. At increasing label shift, SRCC values between EM’s uncertainty estimates and true errors were larger, especially for rLM and hLM CL_int endpoints (Figure ). For LE-MDCK P _app (Figure S5a), fluctuations of SRCC values were observed at higher label shifts, specifically among the top 10–20% of most shifted compounds. Similarly, for the LogP model, EM’s performance gradually improved with label shift until the top 30% compounds with largest shift (Figure S5b).

3.4.3. Discontinuity Shift

The discontinuity shift, quantified according to Diff5NN metric, was previously linked to ADME models’ error in retrospective analyses. For both rLM and hLM CL_int endpoints, correlations between EM’s uncertainty estimates and true model errors improved as the discontinuity shift increased. This trend resembled the behavior observed under label-shift scenarios.

3.4.4. Error Shift

EMs outperformed other UQ metrics in ranking compounds with moderate model errors. However, their advantage diminished for the top 20–30% of test compounds with the highest true errors, as seen in the rLM CL_int, hLM CL_int (Figure ) and LE-MDCK P _app (Figure S5a) endpoints. In these extreme cases, EMs performed comparably to the other UQ methods. While EMs can identify compounds likely to have high prediction error, the most inaccurate property predictions could not be rank-ordered. Even though lower SRCC values observed at higher error levels might reflect a limitation of current UQ metrics, SRCC values are highly dependent on the test data distribution.

Overall, the performance of several UQ metrics (including distance, variance, and EMs) was evaluated across multiple data shifts. These analyses ensure that UQ methods are tested on distributions distinct from those encountered during training, providing a more rigorous and comprehensive assessment of their robustness to different data shift scenarios. Across feature, label, and discontinuity shifts, EM-based uncertainty estimates consistently outperformed other UQ methods. This suggests that explicitly integrating information from both the training set’s chemical space coverage and model outputs improved the robustness of uncertainty estimation across the chemical and property space, including cases of structure–property cliffs. Conversely, EMs cannot accurately rank with the highest prediction errors, showing no significant improvement over standard UQ metrics such as distance to the training set or ensemble variance.

4. Conclusions

In this work, we presented a comprehensive benchmark of UQ metrics for compound property prediction, with a focus on key ADME endpoints relevant to drug design. Leveraging both large in-house and open-source data sets, we conducted realistic evaluations under temporal shifts (prospective application) and enabled future method comparisons.

Our analyses highlighted the value of combining data-based and model-based UQ metrics to capture complementary aspects of prediction uncertainty. Error models integrating chemical space coverage and model-derived outputs, such as predicted value and variance, consistently outperformed standard UQ metrics. Moreover, error models showed strong performance in ranking compounds by prediction error across various endpoints and data sets, while maintaining resilience to data shifts. Since robustness under data shifts is critical for real-world applicability, we designed a benchmarking protocol to evaluate UQ metrics under four types of data shifts: feature, label, discontinuity (property cliffs), and error shifts. This validation setup is applicable even in the absence of temporal information and allows an evaluation under different degrees of shifts, rather than a binary in-distribution vs out-of-distribution classification. Across both temporal and distributional shifts, EMs showed greater robustness than other UQ metrics and served as a more informative indicator of model performance. These findings support the integration of multiple, complementary uncertainty indicators to achieve reliable uncertainty estimates in prospective applications.

While this study emphasizes ranking-based evaluation of UQ metrics by their association with the model’s error, we acknowledge that calibrated UQ metrics may be needed for certain applications. In fact, the UNIQUE framework also incorporates calibration-based and proper scoring rule evaluations, enabling further assessments beyond ranking ability. However, identifying a UQ metric that is a reliable indicator of model error is an essential prerequisite. Building on this, methods such as conformal prediction could be applied to the top-performing UQ metric identified by UNIQUE to generate prediction intervals.

Taken together, the visualizations and metrics presented in this study provide a practical framework for assessing UQ robustness and identifying method limitations. By evaluating UQ methods under diverse and realistic data shift scenarios, this work offers actionable insights for improving uncertainty estimation in ML-driven drug discovery. Future research could further explore the potential of EMs in molecular ML, including their interpretability, their generalizability across broader endpoints and data sets, as well as the evaluation of other architectures and input features.

This work utilizes both proprietary and public data. Proprietary data provides insights into large pharmaceutical data sets, including temporal shifts and applicability to real-world industrial scenarios. Open-source data and code were also used and are available to ensure reproducibility of our results. The public ADME data set used in this study is available at DOI: 10.1021/acs.jcim.3c00160. The code for model building is available at DOI: 10.1021/acs.jcim.3c01250. The uncertainty quantification benchmark was performed using the UNIQUE framework (previously published at JCIM by current coauthors JL and RRP et al.) and available at DOI: 10.1021/acs.jcim.4c01578. The GitHub repository is available at https://github.com/Novartis/UNIQUE.

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acs.jcim.5c02381.

• Additional model details (assays, tasks, architecture); specifications of the computational environment; metrics for UQ evaluation; distributions for in-house data; and data shift results (PDF)

RRP and JL conceived the study; RPP carried out the analyses; RRP and RPP wrote the manuscript; all authors planned the analysis, discussed the results, examined the manuscript and agreed to the final content.

The authors declare no competing financial interest.