Development and Evaluation of Conformal Prediction Methods for Quantitative Structure–Activity Relationship

Abstract

The quantitative structure–activity relationship (QSAR) regression model is a commonly used technique for predicting the biological activities of compounds using their molecular descriptors. Besides accurate activity estimation, obtaining a prediction uncertainty metric like a prediction interval is highly desirable. Quantifying prediction uncertainty is an active research area in statistical and machine learning (ML), but the implementation for QSAR remains challenging. However, most ML algorithms with high predictive performance require add-on companions for estimating the uncertainty of their prediction. Conformal prediction (CP) is a promising approach as its main components are agnostic to the prediction modes, and it produces valid prediction intervals under weak assumptions on the data distribution. We proposed computationally efficient CP algorithms tailored to the most widely used ML models, including random forests, deep neural networks, and gradient boosting. The algorithms use a novel approach to the derivation of nonconformity scores from the estimates of prediction uncertainty generated by the ensembles of point predictions. The validity and efficiency of proposed algorithms are demonstrated on a diverse collection of QSAR data sets as well as simulation studies. The provided software implementing our algorithms can be used as stand-alone or easily incorporated into other ML software packages for QSAR modeling.

Introduction

Quantitative structure–activity relationship (QSAR) regression models are routinely applied in the drug discovery process for predicting the biological activities of molecules from the molecular structure-based features. The predictions are used to prioritize candidate molecules for future experiments and help chemists gain better understanding of how structural changes affect activities.¹⁻³ While most of the previous efforts focus on improving the accuracy of point predictions, quantifying the uncertainty in the prediction will add valuable insights.⁴⁻⁸ For regression tasks, prediction intervals (PIs) are often used as quantitative measures of the reliability or confidence in the point prediction at a given probability. A well-calibrated PI contains future observations with a prespecified probability, which is called nominal coverage. The width of a calibrated PI gives users an intuitive estimate of the precision of the prediction, with a wider interval indicating less precision (more uncertainty) than a narrower interval. For example, an interval of 2.0 to 3.0 at 95% probability for compound A means that the true value of the activity has a 95% chance of falling within 2.0 and 3.0. In comparison, a wider interval of 1.0 to 4.0 at 95% for compound B would indicate that B is less reliably predicted than A.

There are various methods for estimating prediction uncertainties. Some methods directly estimate the quantile or variance of prediction errors, while others provide relative uncertainty scores between different molecules, which requires further calibration for obtaining valid PIs.⁶ Quantile regression methods⁹⁻¹¹ have been explored with various machine learning (ML) algorithms, including neural networks,¹² random forests (RFs),¹³ and light gradient boosting (GB) machine;¹⁴ however, the PIs constructed from the quantile estimates often lead to an under-coverage, i.e., the fraction of true activities falling within the intervals is smaller than the specified nominal coverage. Bayesian methods estimate the posterior distribution of molecular activity given a molecular structure. Thus, the PIs could be computed using the variance of the prediction errors under certain assumption of their distribution. An example of this approach is studied in Feng et al.,¹³ the Bayesian additive regression trees that simultaneously estimate molecular activity and the error variance as a measure of prediction uncertainty, under the assumption that the errors are normally distributed. This assumption, however, is often violated in many QSAR applications.¹⁵⁻¹⁷ A practical concern with some methods is the high computational cost, which is especially high for Bayesian methods. Hirschfeld et al.⁶ benchmarked several uncertainty quantification methods that are applicable to neural networks, including ensemble-based methods,^18,19 distance-based methods, mean-variance estimation,²⁰ and union-based methods.²¹ This work recommended several top-performing methods that produced an estimate of the predicted error variances with the lowest negative log likelihood under the Gaussian distribution assumption for prediction errors. The likelihood-based evaluation metric, however, is highly sensitive to the distributional assumption. As was mentioned above, prediction errors in many QSAR applications are non-Gaussian.¹⁵⁻¹⁷ The top recommended methods in Hirschfeld et al.⁶ are union-based methods involving training a neural network for point prediction, and a second model, for example, the Gaussian Process or RF regression to estimate the variance of errors in prediction produced by the first model. Training of two models typically requires substantially more data. Comprehensive reviews on uncertainty quantification methods and/or applications could be found in Mervin et al.,⁷ Abdar et al.,²² Tian et al.,²³ and Yu et al.⁸ And some additional recent work in this field could be found in Alvarsson et al.,²⁴ Bostrom et al.,^25,26 and McShane et al.²⁷

We are seeking practical methods to provide PIs accompanying the point predictions for large QSAR data sets in an industrial-scale pharmaceutical drug discovery environment. A good algorithm should meet the following requirements:

• (1)Marginal validity: The empirical marginal coverage for a test set is the proportion of PIs that contains these molecules’ true activity measurements. Validity means that the empirical coverage must match the user-specified (nominal) coverage. Marginal, in this context, is understood as averaged over all molecules not used in model development.
• (2)Conditional validity (also called informativeness or adaptivity): The conditional coverage, i.e., PI coverage for each molecule, should be close to the nominal coverage. In other words, the width of the PI should closely follow the uncertainty in the prediction of any specific molecule. Under the homoscedastic conditions where the variance of prediction errors does not depend on the molecular descriptors, PIs for all molecules will have the same width. In contrast, under heteroscedastic conditions, where the variance depends on the descriptors, the width of the PI should increase or decrease in accordance with the variance. Compared to marginal validity, conditional validity is a much more stringent requirement usually very difficult to fully achieve. However, we still aim to have PIs with widths adaptive to the prediction errors, in order to approximate this property.
• (3)Efficiency or tightness of PIs: The width of the PIs should be as narrow as possible.
• (4)Low computational cost: The computation of PIs should not involve a significant computational effort over generating the original point prediction model. In real application, it is crucial to ensure the computational efficiency, since the QSAR models need to be rebuilt regularly with accumulated experimental data.
• (5)Independence of model or data distribution: The properties of the PIs, such as validity, efficiency, and adaptivity, do not depend on the choice of ML methods or any assumption about the distribution of prediction errors.

Many evaluation metrics of PIs have been explored in previous benchmark studies, and several recommendations have been proposed.^6,23,28,29 Although it is difficult to find a single method that is superior to others under all criteria, the conformal prediction (CP) framework could satisfy most of these practical considerations and has attracted increased attention in recent QSAR applications.^{18,19,30−35} The PIs generated by CP methods are guaranteed to achieve valid marginal coverage, i.e., on average, the test set PIs to contain the measured molecular activities with a user-specified probability, under the assumption of data exchangeability, i.e., that the training set molecules and molecules being predicted are random selections from the same pool of molecules.^36,37 This marginal validity property does not depend on any distributional assumptions or model assumptions and holds for any sample size.³⁶ The CP is applied as a companion to any pretrained model for a QSAR regression task, and many conformal algorithms require little extra computational costs besides the training of the original prediction model. Construction of conformal PIs is based on a “nonconformity” score that quantifies how unusual a data point is relative to the training data.³⁸ We prefer to use nonconformity scores that do not require additional modeling efforts. The adaptivity property of PIs depends on the choice of a nonconformity score. A theory for selecting the nonconformity score is yet to be developed, and the choice needs to be evaluated with empirical studies. Several computationally efficient CP algorithms have been developed for QSAR regression;^18,19,34 however, a comprehensive evaluation of their properties, especially their adaptivity, i.e., ability to handle heteroscedasticity of prediction errors, remains to be done.

In this work, our first contribution is to develop a computationally efficient CP algorithm, named adaptive calibrated ensemble (ACE) that is based on a specifically designed nonconformity score. The ACE algorithm can be used with any model that provides both point prediction and an uncertainty estimate for each molecule. For any ensemble-based model, the mean prediction over the ensemble is the “prediction” (i.e., the point estimate) and the standard deviation of predictions can be used as the prediction uncertainty estimate. The point estimate and prediction uncertainty estimate are inputs for ACE. The ACE nonconformity score is computed from the point prediction and the data-driven transformation of the prediction uncertainty score. If the uncertainty scores effectively represent the relative uncertainty between molecules, the ACE PIs would achieve not only a prespecified marginal coverage but also an approximate conditional coverage.

Deep neural networks (DNNs) and GB models are among the most predictive descriptor-based ML methods.³⁹⁻⁴¹ Our second contribution is proposing novel approaches for generating uncertainty estimates for these methods, named DNN-multitask and GB-tail, respectively. DNN-multitask is compared to the state-of-the-art DNN-dropout method for generating uncertainty estimates from ensembles.^8,42

Our third contribution is a comprehensive analysis of a diverse collection of real QSAR data sets, with special attention to the analysis of conditional validity and efficiency.

Conditional validity and adaptivity of PIs in the heteroscedastic case is one of our work’s primary objectives and that of other researchers. The proposed methods include modifying the nonconformity scores by normalizing them to the conditional measures of prediction uncertainty. These methods and relevant references are discussed later. The method proposed in our paper also uses conformity score modification in a novel way, efficiently handling heteroscedasticity that is demonstrated with a large volume of diverse QSAR data. Different approaches to handling heteroscedasticity or, in general, obtaining PIs satisfying conditional validity have been proposed recently and are based on quantile regression,⁴³ conditional histograms,⁴⁴ and CP distributions.^45,46 To our knowledge, these approaches have yet to be used in the QSAR literature and are currently being investigated by the authors of this paper.

The paper is organized as follows. The Data Sets and Molecular Descriptors section describes QSAR data sets and their molecular descriptors. In the Methods section, we briefly review the CP framework and applications to QSAR, introduce the proposed ACE CP algorithm, define evaluation metrics, and explain in detail how to obtain ensemble predictions or raw prediction uncertainty scores for several ML algorithms. The Results section presents the results of applying ACE jointly with several popular QSAR predictive algorithms and shows the validity, efficiency, and adaptivity of proposed methods on diverse QSAR data sets.

Data Sets and Molecular Descriptors

Two collections of QSAR data sets are used in this study:

• ChEMBL: IC₅₀ data sets for 23 diverse protein targets and receptors from the ChEMBL database. The data sets were obtained from Cortes-Ciriano et al.,¹⁸ excluding the smallest data set “A2a” with only 203 molecules. We generated molecular descriptors according to the procedures described in Cortes-Ciriano et al.¹⁹ The circular Morgan fingerprints⁴⁷ were computed using RDkit⁴⁸ (version 2021.03.2) with the radius set to 2, and the output fingerprint length was 2048.

• Kaggle: The 15 QSAR data sets used in the 2012 “Merck Molecular Activity Challenge” Kaggle competition and released in Ma et al.³⁹ These data sets are of various sizes for either on-target potency, off-target activity, or absorption, distribution, metabolism, and excretion (ADME) properties. The molecular descriptor is the union of the “atom pair” (AP) descriptors from Carhart et al.⁴⁹ and “donor–acceptor pair” (DP) descriptors.⁵⁰ Each data set was provided in two parts: the time-split training set and test set. In this work, we only use the training set in each data set, since investigating the covariate-shift problem for time-split test set is out of scope for this study.

Compared to the ChEMBL collection with all IC₅₀ data sets, the Kaggle collection contains a mixture of tasks (including on-target potency or off-target absorption, distribution, metabolism, and excretion activities), larger data sets with higher dimension of molecular features, and examples of activity distributions far from Gaussian.

Due to the complex nature of QSAR data sets, the distribution of prediction errors is often unknown. Some data sets may have near-constant prediction error variance that is unrelated to molecular features. In this case, applying constant width PIs for all molecules (i.e., homoscedastic) is preferred. Other data sets may have varying prediction error variance for different molecules, for example, variability of experimental measurements increases with their magnitude. In this case, PIs with varying widths correlated with prediction errors (i.e., heteroscedastic) are desirable. A good algorithm should provide adaptive PIs suitable for both cases. The data sets we use here are likely to contain both situations.

Methods

Conformal Prediction

CP is a model-agnostic framework for measuring prediction uncertainty.^36,38 In regression tasks, the PIs constructed by CP achieve valid marginal coverage with the general i.i.d. (independent and identically distributed) data assumption.^37,51 The original CP algorithm, which is called transductive CP, is computationally expensive since it requires training a new predictive model for every additional molecule. For large data sets in QSAR problems, a modified version of the CP algorithm called inductive CP (ICP) or split CP has been used.^{18,19,33,34,52} The steps involved in ICP are as follows:

• (1)Specify an ML method to generate a model that provides point predictions.
• (2)Define a nonconformity measure to quantify how unusual the prediction error of a molecule is compared to the others. The nonconformity measure may depend on the prediction ŷ or the descriptor vector X of a molecule.
  • For constructing homoscedastic PIs, which have a constant width for all molecules, the nonconformity measure is defined as the absolute value of residuals

    
  • For constructing heteroscedastic PIs that apply to individual molecules, the nonconformity measure can be defined as the normalized absolute value of residuals

    1

    where σ(X) is an estimate of the precision of the point predictor.⁵³ The homoscedastic nonconformity measure is corresponding to a special case when σ(X) ≡ 1.
• (3)Divide the available training data randomly into a “proper training set” and a “calibration set”. For QSAR tasks, the relationship between molecular structure descriptors and activities is usually highly complex. It is preferable to use a large fraction of data, e.g., around 80%, as the “proper training set”, in order to train an accurate point predictor model.
• (4)Use the proper training set to construct a model, and use the model to predict the activity of molecules in the calibration set. Calculate the nonconformity score α_j for each calibration set molecule x_j (1 ≤ j ≤ n).
• (5)Specify a desired nominal coverage probability θ, and calculate the θ^th percentile of nonconformity scores in the calibration set as α_θ.
• (6)
  The PI for a new molecule x_new in the test set would be

  

The choice of the scaling factor σ(X) in nonconformity measure affects the efficiency of CP (i.e., the width of PIs).³⁸ There are two typical approaches to define the scaling factor used in previous studies: model-based^33,43,52,53 and ensemble-based approach.^18,19,34 In this paper, we will use the ensemble-based approach. Here, σ(X) is a function of the ensemble prediction uncertainty s(X), which is the standard deviation of predictions from an ensemble of predictions. Below, we abbreviate σ(X) and s(X) as σ and s , respectively, when they are clearly dependent on molecular descriptors X. Ensemble-based scaling factors performed slightly better than the model-based scaling factors in the previous work on QSAR data sets.³⁴ Some ML methods like RFs natively produce an ensemble of predictions. Other popular ML methods such as DNN or boosting need modification to produce an ensemble of predictions, preferably without being too computationally expensive. This is discussed in the ML Algorithms section.

ACE Conformal Predictor

We need to transform s, the ensemble prediction uncertainty, to the scaling factor σ in eq 1. The exponential transformation of s, i.e., σ = e^s, has been a popular choice of the scaling factor in previous studies.^18,19,34,54 We call this scaling method “expSD”. Svensson et al.³⁴ explored different transformations σ = e^γ × s with various weight γ ranging from 0.05 to 1.25 and concluded that the best average performance was obtained when γ = 1. However, in our experience, this scaling method is not always optimal for different data sets or ensemble approaches. The range of scaling factor σ affects the range of PI widths in a data set. For ensemble methods that produce smaller s, the exponential transformation will lead to near-constant scaling factor σ and thus almost constant-width PIs, which may not be desirable if the data set is intrinsically heteroscedastic. On the other hand, it may also lead to excessively wide and noninformative PIs for some molecules.

Therefore, we proposed a flexible calibration algorithm, ACE, which will generate transformation of any raw prediction uncertainty score s as a scaling factor for nonconformity scores adaptive to different data sets and/or ensemble models. The score s can represent the variability of predictions in an ensemble of models, or any relative uncertainty scores that correlate with the prediction errors. The ACE algorithm finds the optimal transformation by minimizing the coverage error conditional on the PI width via repeated cross-validation on the calibration set. The major steps in ACE are as follows:

• (1)Calculate the mean μ_s and standard deviation sd_s of the s values on the calibration set, and calculate the normalized s as .
• (2)
  Define b as the average of the absolute error in the calibration set. And define the scaling factor σ as a function of the parameter a

  

  where 0 ≤ a ≤ b to ensure that σ(a) is always a positive and nondecreasing function of s̃.
• (3)Perform repeated twofold cross-validation on the calibration set, and use grid-search to find the optimum value of a to achieve the lowest average coverage error over four equal size subgroups defined by PI widths.

More details are provided in the ML Algorithms section below.

ML Algorithms

In this section, we describe the supervised learning algorithms commonly used in QSAR applications and how to generate computationally efficient prediction uncertainty score s, as illustrated in Figure 1. Additional details of implementation and hyperparameter settings of each algorithm are provided in the Supporting Information.

Figure 1.

Illustration of generating point predictions and prediction uncertainty scores using various ML algorithms.

Random Forests

RF has been a very popular QSAR method due to its high prediction accuracy and robustness to the choice of hyperparameters.⁵⁵ RF is an ensemble of independent decision trees. The average of individual tree predictions is the final prediction for a molecule; and s(X) is the standard deviation of tree predictions of the molecule represented by X. This has been an effective measure of prediction uncertainty in previous conformal applications.^18,19,34

Compared to other supervised learning methods, RF has a unique appealing feature: the out-of-bag (OOB) data already represents a random split, and one does not need to make an explicit “proper training set/calibration set” split.⁵³ This prediction uncertainty estimation strategy is referred to as “RF-OOB” in the later sections.

Deep Neural Networks

DNN is a practical QSAR method for large data sets that usually achieve superior predictive performance,³⁹ although it is somewhat computationally expensive, sometimes requiring GPU computing hardware. Here, we consider only fully connected descriptor-based DNNs.

We considered two approaches for generating ensemble predictions that require training only one DNN prediction model: the “DNN-dropout” as it is called in previous reports^19,42 and a novel “DNN-multitask” method.

DNN-dropout: Cortes-Ciriano and Bender¹⁹ built conformal predictors for the ChEMBL data sets using test-time dropout⁴² ensembles, where the ensemble prediction was created by allowing random dropout of DNN hidden layer nodes and repeating the prediction process 100 times. The average and standard deviation across the 100 repeated predictions for a molecule were used as its predicted value and a measurement of the prediction uncertainty, respectively. We adopted the same DNN structure and optimization algorithm described by Cortes-Ciriano and Bender¹⁹ for the ChEMBL data sets. In the study by Cortes-Ciriano and Bender,¹⁹ multiple dropout rates, 0.1, 0.25, and 0.5, were explored, and the results demonstrated that the performance for different dropout rates was comparable. We adopted the median value of the dropout rate, i.e., using a constant dropout rate of 0.25 in all hidden layers. For Kaggle data sets, we used the recommended parameter setting from Ma et al.³⁹

DNN-multitask: We created an artificial sparse multitask DNN model from a single-task QSAR data set, by replicating the molecular activities K times with random omissions of molecules (with the probability of omission p), for example

where the artificial vector outcome for each molecule i is a K-dimensional vector with elements

“NA” represents an omitted molecular activity. The average of multitask outputs is taken as the point prediction. The standard deviation of multitask predictions for each molecule is used as the prediction uncertainty metric s(X). The intuition is that molecules that are “easier” to predict would get more consistent predictions across this pseudoensemble model. To explore whether the multitask standard deviation is sensitive to the choice of hyperparameters K and p, we evaluated multiple combinations of (K, p) pairs via simulations (see the Supporting Information) and recommend using a higher omission probability (p = 0.6) and a moderate number of the output nodes (K = 20 or 50).

For Kaggle data sets, the neural network structure and optimization algorithm are the same as Ma et al.,³⁹ except for the output layer size. For the ChEMBL data sets, since the dimensions of input molecular features are smaller and the molecules are “easier” to predict, we used a smaller neural network structure, which achieved similar prediction accuracy with less computational cost compared to the DNN-dropout setting in the study by Cortes-Ciriano and Bender.¹⁹

Gradient Boosting

GB is a widely used QSAR method due to its computational efficiency and accuracy.^40,41 A GB model consists of a series of many (typically 1000 or more) shallow decision trees. The prediction of the model on a molecule is the sum of predictions of the decision trees on that molecule. The trees are added to the model in iterations such that the errors from the current model are used to grow a new tree, so that the new tree is forced to learn information that the current model did not yet learn from the data. Thus, in general, we expect the trees learned earlier in the sequence to have larger contributions to the overall prediction than those later in the sequence. For prediction of a new molecule, if the quality of the prediction is high, we would expect that the absolute magnitude of the contribution would diminish for later trees. If the absolute contribution falls to nearly zero after the first few trees, it indicates that the prediction for that molecule is reliable. On the other hand, if the absolute contributions from the last few trees are still large, then the prediction is less likely to be reliable. Therefore, we propose the “GB-tail” method: for each molecule, let s be the mean absolute value of the contributions from the final w fraction of trees (for instance, for w = 0.2, the last 200 out of 1000 trees). This approach requires only postprocessing of tree predictions and has a minimal computational cost.

To check whether the estimate of the prediction uncertainty by the GB-tail method is sensitive to w, we investigated the impact of varying the hyperparameter w (Supporting Information) and recommend w = 0.2.

Evaluation Metrics

We compared three types of conformal PIs: Homoscedastic PIs (abbreviated as “homo” in the figures), heteroscedastic PIs with the “expSD” scaling factor (expSD), and heteroscedastic PIs with the scaling factor computed from the proposed ACE algorithm (ACE). They are applied in combination with four supervised learning algorithms/ensemble schemes introduced in the ML Algorithms section.

We evaluated several important aspects of the prediction models, including the accuracy of point predictions, computational costs, and the informativeness of PIs. All the evaluation metrics are calculated on the test set and averaged across multiple random repeats and/or data sets.

The prediction accuracy is measured by the squared Pearson correlation coefficient (R-squared) and the normalized root-mean-squared error (RMSE). These metrics serve the purpose of assessing the quality of predictions generated by different ML methods. It is essential to ensure satisfactory prediction accuracy before evaluating the performance of conformal PIs.

The performance of a conformal predictor is usually evaluated in terms of validity and efficiency. A valid PI has a coverage probability no less than the prespecified nominal coverage level, i.e., if the probability is 90%, then at least 90% of the true activity measurements should fall within the intervals. The efficiency is measured by the widths of the PIs. In regression analysis, a conformal predictor is considered favorable if it produces narrow PIs around the point predictions while maintaining the desired nominal coverage. ICP algorithms provide PIs that guarantee marginal validity regardless of the underlying data distribution. However, the efficiency of these PIs can be influenced by the choice of nonconformity scores.^38,51 The selection of an appropriate nonconformity score is crucial in achieving optimal efficiency.

Coverage error: For each nominal coverage probability, we calculated the coverage error as the difference between the actual coverage of test set compounds and the nominal coverage. For example, if the nominal coverage of the interval is 95% and the actual coverage of the interval is 85%, the difference is 10%. Although the conformal approach theoretically guarantees that the coverage is correct, in practice, coverages of finite samples will fluctuate around the nominal value.³⁶ We also calculated the mean absolute error (MAE) of the test set coverage across different data sets and repeats.

Efficiency: To compare the efficiency across algorithms and data sets, one needs to normalize the raw PIs. One way to do this is to find the ratio of the width of the raw interval to the width of the interval in the corresponding “no-model” case. The no-model case uses the distribution of observed training set activities rather than predictions. For example, for a nominal coverage of 90%, the interval is between the fifth-percentile and 95th-percentile values of the observed activities in the training set. The “no-model” PI should be wider than the PIs obtained by the ICP analysis where the model-based point predictions are used. When comparing the efficiency between heteroscedastic and homoscedastic PIs, we scaled the heteroscedastic PI widths by the corresponding homoscedastic PI width for the same test set and the same point prediction model.

Adaptability: The ability to differentiate easy-to-predict and difficult-to-predict molecules, i.e., “adaptability”, is another top consideration for constructing informative PIs.⁵⁶⁻⁶⁰ The expectation is that molecules that can be more accurately predicted have narrower PIs. Ideally, one would like the PI to cover the true activity with the prespecified nominal coverage probability for every molecule, i.e., to achieve conditional validity. However, this cannot be achieved without imposing assumptions on the data distribution.⁵⁸ Although there is no theoretical guarantee for ICP algorithms to satisfy the conditional validity, one may approximate this property with heteroscedastic PIs constructed from well-designed nonconformity scores. We used a size-stratified coverage metric to evaluate the adaptivity of PIs in the following way. First, we sorted the test set molecules by their PI widths, and divided them into five equal-size subgroups. We then calculated the coverage error within each subgroup, as well as the average absolute error of coverages across five subgroups for each data set. If the coverage in these subgroups significantly deviates from the nominal coverage, it indicates a lower quality of the conditional coverage.^36,56,57,61

In the Supporting Information section, we use simulated data with Gaussian-distributed noise that have either constant variance or changing variance to demonstrate how the state-of-the art expSD and the proposed ACE CP algorithms performed in these cases. Based on the simulation study, the expSD method cannot provide PIs adaptive to the magnitude of errors and does not perform well for all ML algorithms. When applying expSD with the DNN-multitask and GB-tail ML method, the PIs have near-constant widths and are not adaptive to the errors from different molecules. The reason is that the raw prediction uncertainty scores from DNN-multitask and GB-tail span a much narrower range and it leads to close to constant scaling factor σ in the expSD method. And the results from expSD with RF-OOB or DNN-dropout methods are also unsatisfactory: For data simulated with constant variance, they have larger coverage errors in size-stratified subgroups than those from either homo or ACE and are very likely to produce some PIs that are too wide to be useful. Thus, we decided to exclude the expSD CP algorithm from any further comparisons in the application section.

Results: Application to ChEMBL and Kaggle Data Sets

In all the following numerical experiments, each data set is randomly split into a proper training set (70% of the data) for training a prediction model for molecular activity and generating raw uncertainty scores; a calibration set (15%) for CP; and a test set (15%) for evaluation. The data split is repeated 20 times, and the same splits are used for all algorithms. For RF models, since we use the OOB data for CP instead of a separate calibration set, the union of the proper training set and calibration (85% of the data) is used for RF model training, and the same 15% test set is used for evaluation.

The average predictive performance and computational time over repeats for two groups of data sets are listed in Table 1. All models achieved satisfactory prediction accuracy. The two DNN models and the GB model perform slightly better than RF, which is usually used as the baseline method in QSAR modeling, despite that the RF model uses more training data (85% of an entire data set) compared to others (70% of an entire data set). Also, the DNN-multitask and GB models took less time compared to DNN-dropout and RF. While the accuracy and run-time of point predictors may not directly affect the performance of conformal predictors, these comparisons demonstrated the necessity of developing suitable uncertainty quantification methods tailored for various practical QSAR models, in addition to the widely used RF-based conformal predictors.

Table 1. Summary of Test Set Predictive Performance and Run Time (in Seconds) of Each ML Method for ChEMBL and Kaggle Data Sets^a.

	ChEMBL			Kaggle
ML prediction algorithm	R-squared	RMSE	runtime (s)	R-squared	RMSE	runtime (s)
RF	0.633 (0.115)	0.710 (0.086)	41.0	0.683 (0.084)	0.565 (0.073)	1314.3
DNN-dropout	0.652 (0.112)	0.688 (0.085)	127.8	0.703 (0.106)	0.541 (0.098)	893.4
DNN-multitask	0.645 (0.113)	0.697 (0.088)	36.2	0.700 (0.108)	0.547 (0.102)	781.3
GB	0.644 (0.114)	0.694 (0.087)	8.4	0.706 (0.098)	0.536 (0.089)	143.8

^aThese numbers represent an average over the data sets and repeats. The standard deviation for the R-squared and RMSE is provided in brackets. The RMSE of Kaggle data sets is scaled by the standard deviation of molecular activities.

We created conformal predictors using two CP algorithms (homo and ACE) and four ML methods (RF-OOB, DNN-dropout, DNN-multitask, and GB) under eight nominal coverage levels ranging from 60 to 95% with increments of 5%. Figure 2 shows the boxplots of the marginal coverage error on test sets under different nominal coverage levels for each method and CP algorithm. In all cases, the coverage errors are centered around zero, which indicates that they all achieved marginal validity. Since the Kaggle data sets are larger than ChEMBL data sets, the distributions of errors have less variability. The comparison of variability in marginal coverage errors between the ML method and CP algorithm is shown in Figure 3. For each ML method and CP algorithm, we averaged the absolute value of coverage errors across data sets and repeats at multiple nominal coverages. The RF-OOB method has lower MAEs of marginal coverage at all nominal levels. The RF model is trained on all available data without splitting into proper training sets and calibration sets, and it used the out-of-bag data for calibration, which is effectively larger than the stand-alone calibration sets in other methods. Due to this unique feature of RF, it achieved more robust performance in the marginal coverage. There was no difference between ACE and homoscedastic, which is expected.

Figure 2.

Boxplots of marginal coverage errors in two groups of data sets for eight nominal coverages ranging from 60 to 95%. Top row: ChEMBL data sets; bottom row: Kaggle data sets. In all the boxplots, the horizontal line drawn in the middle of each box denotes the median, and the range of box is from 25th percentile (Q₁) to 75th percentile (Q₃). The dots represent data beyond 1.5 times the interquartile range (Q₃ – Q₁) above Q₃ or below Q₁.

Figure 3.

Absolute marginal coverage errors averaged across data sets and repeats, for eight nominal coverages ranging from 60 to 95% and two groups of data sets. Lower values indicate better marginal validity. Top row: ChEMBL data sets; bottom row: Kaggle data sets.

Figure 4 compares the efficiency of conformal predictors. In Figure 4a, the DNN-dropout, DNN-multitask, and GB-tail methods have slightly narrower average PI widths than the RF-OOB method in the lower nominal range (60 to 85%), but the distributions of their average PI widths have a wider spread. Also, there are a few outliers at the higher nominal level of 95%. Figure 4b compares the efficiency between ACE and homoscedastic conformal algorithms. For each ML method, the average PI width of the ACE algorithm of each test set is scaled by the corresponding homoscedastic PI width. As the nominal level increases, the ACE algorithm is more likely to produce narrower PIs, on average, compared to homoscedastic.

Figure 4.

Boxplots of the scaled average PI widths for eight nominal coverages ranging from 60 to 95%. For each ML algorithm, CP method, and each of the 20 repeats, the PIs for each nominal coverage are averaged across the test set. In order to facilitate the comparison of average prediction widths across different data sets in the boxplots, it is necessary to scale the PIs. The two subfigures below illustrate two different scaling approaches. (a) The PIs are scaled by the “no-model” interval width, which is calculated from the training set activities rather than model predictions. For a nominal coverage of (1 – α), the “no-model” interval is between the α/2 quantile and (1 – α/2) quantile values of the observed activities in the training set. Top row: ChEMBL data sets; bottom row: Kaggle data sets. Each column of subplots corresponds to one nominal coverage level. (b) The PIs are scaled by the corresponding homoscedastic PI width for the same test set and the same point prediction model. Top row: ChEMBL data sets; bottom row: Kaggle data sets. The x-axis is the nominal coverage level.

Figure 5 demonstrates the conditional coverage property of ACE in contrast to homoscedastic using the nominal level of 80% as an example. Again, each test set was sorted by the PI widths for each method in increasing order and split into five equal-size subgroups. Similar to what we observed in the simulation study, the coverage of the ACE algorithm is closer to the expected nominal coverage (dashed line) in all subgroups compared to homoscedastic. In most situations, except the DNN-dropout model for ChEMBL data sets, there is a clear decreasing trend of homoscedastic PI coverage from the first subgroup to the fifth subgroup. It indicates that most of the molecules with smaller PIs have smaller prediction errors, which leads to overcoverage of homoscedastic PIs in the first two subgroups. Also, the molecules in the last two subgroups have larger prediction errors than the PI width and hence this results in the under-coverage of homoscedastic PIs. In other words, the decreasing trend of homoscedastic PI coverage demonstrates that the PI effectively differentiates between the molecules with small prediction errors from those with large prediction errors. Thus, the PIs from DNN-dropout ensembles in some cases are not as informative as the other methods. Figure 6 compares the conditional coverage of ACE PIs generated by different methods. For ChEMBL data sets, the DNN-dropout model shows a higher MAE of coverage by subgroups at multiple nominal levels. For some Kaggle data sets, the DNN-dropout model also produced noticeably higher errors at lower nominal levels.

Figure 5.

Test set coverage of subgroups defined by PI widths, at a nominal level of 80%. For each ML algorithm, data set, and repeat, the test set molecules are sorted by the PI widths in increasing order and split into five equal-size subgroups numbered 1 to 5. And the test set coverage for each subgroup using different CP methods is calculated as the proportion of true activity that falls within the PI at a nominal level of 80% (horizontal dashed line).

Figure 6.

Comparison of the conditional coverage of ACE PIs generated by different methods. For each ML algorithm, data set, and repeat, the test set molecules are sorted by the PI widths in increasing order and split into five equal-size subgroups. The test set coverage for each subgroup using the ACE CP method is calculated as the proportion of true activity that falls within the PI at various nominal levels. The y-axis is the MAE of test set coverage across subgroups, i.e., the absolute errors of test set coverage averaged across five subgroups in each test set, with lower values indicating better conditional coverage.

Figure 7 shows the ACE PIs using the Kaggle 3A4 data set as an example. The molecular activities in the 3A4 data set have a truncated distribution causing heteroscedastic prediction errors. In each subplot, on the horizontal axis, the test set molecules are ordered by the PI widths from the corresponding ML method. The orange line shows that the spread of the absolute errors is increasing with the PI width for all four methods, indicating a strong association between the magnitude of the true prediction error and PI width. Thus, with the ACE algorithm, we obtained well-calibrated heteroscedastic PIs with the widths adaptive to the true absolute prediction errors. Empirical coverage of the ACE PIs reflected by the proportion of points below the regression line (numbers are not shown) is close to the nominal levels at each interval width. Thus, the ACE method provides meaningful estimates of the prediction errors.

Figure 7.

Association of the ACE PI widths and absolute prediction errors (nominal level of 80% for the Kaggle data set 3A4, and shown results are for one test set out of 20 repeated runs). The orange line is the PI width from the ACE algorithm. Each dot in the scatter plot is a test set molecule, colored by whether it is successfully covered by the PI (with the absolute error below the orange line). Both absolute errors and the PI width are scaled by the corresponding homoscedastic PI width. The horizontal black dash line with y-intercept at 1.0 represents the scaled homoscedastic PI size (unit length).

Discussion

In this work, we developed the ACE algorithm, an inductive conformal predictor whose nonconformity scores are calculated using estimates of the prediction uncertainty generated by the ensemble of point prediction models. We evaluated this method using both simulated and large collections of real QSAR data. The ACE algorithm produced PIs that are well adapted to the data: for homoscedastic data, the interval width is nearly constant; and for heteroscedastic data, the width varies with the magnitude of prediction errors and achieves close to nominal coverage for each molecule. This adaptiveness feature of ACE PIs makes it highly informative in QSAR applications. One can use the interval size to differentiate the molecules which are accurately predicted vs those that are not.

The ACE algorithm is applicable to any prediction model, as long as a raw prediction uncertainty score can be computed, and that uncertainty correlates with the variance of actual prediction errors. Since the PI widths produced by the ACE algorithm as well as other ensemble-based CP algorithms are positively correlated with the raw prediction uncertainty scores, it is critical to develop ML methods that produce raw uncertainty scores that effectively track the errors to achieve adaptivity. Some of the widely used QSAR methods, such as DNNs, provide highly accurate point predictions but do not generate raw uncertainty scores because they do not naturally produce ensembles. The GB method, although an ensemble method, also does not natively generate a prediction uncertainty estimate. To address this, we proposed the novel ensemble DNN-multitask and GB-tail methods, to compute the raw prediction uncertainty score from DNN and GB point prediction models with a small computational cost. The ACE PIs with raw prediction uncertainty scores obtained by both these methods showed competitive performance in a diverse collection of QSAR data sets. Notably, in the case of neural net models, DNN-multitask has shown to produce more useful ensembles compared to DNN-dropout. Additionally, implementing the DNN-multitask method involves only a simple reformatting of the training molecular activities without changes in the neural net structure. This method could be also applied to the convolutional neural network⁶²⁻⁶⁴ or graph convolutional networks.⁶⁵⁻⁶⁸ By tailoring uncertainty quantification techniques to different ML methods, we can enhance the applicability of conformal predictors in real-world scenarios. While a comparison across different ML methods could provide valuable insights, it falls beyond the scope of this paper.

The topic of uncertainty quantification has also been explored in the applicability domain (AD) research, a subfield in QSAR.⁶⁹⁻⁷¹ However, there are many definitions of the concept of AD and there is no consensus on the best approach.^2,32 To improve the value of AD model, Hanser et al.⁷² clarified, formalized, and extended the definition of AD. In this work, the evaluation of AD was split into three well-defined subtopics: the chemical structure space boundaries of model applicability, the reliability of predictions, and whether the predictions can help to make a clear decision. The well-defined mathematical framework of CP produces PIs with straightforward interpretation, which is a valuable addition to the prediction reliability assessment methodology in AD.

Although this work focuses on developing conformal PIs for QSAR regression tasks, there are CP algorithms for classification modeling and have been applied in QSAR before.^{24,32,33,73−75} In classification, the nonconformity score is usually defined based on the predicted probabilities associated with each class. The proposed DNN-multitask and GB-tail methods can also be used to compute the raw prediction uncertainty scores for classification models, which could be helpful for constructing novel nonconformity measures. Another interesting direction to explore in future study is comparing the performance of uncertainty estimation methods on various tasks, either regression or classification, categorized by the underlying data distribution.

In conclusion, the conformal predictors computed by the proposed ACE algorithm jointly with highly accurate and commonly used ML models may serve as practical uncertainty quantification tools for QSAR modeling, producing accurate and informative PIs without compromising the point prediction quality or computational efficiency.

Supporting Information Available

The Supporting Information is available free of charge at https://pubs.acs.org/doi/10.1021/acsomega.4c02017.

• Visualization of data set distribution, download links for two collections of QSAR data sets, ML algorithm implementation and parameters, hyperparameter tuning for prediction uncertainty score methods, simulation settings, results for data with simulated noise, and companion summary statistic tables for boxplot figures in section “Results: Application to ChEMBL and Kaggle data sets” (PDF)

The authors declare the following competing financial interest(s): All authors are employed by Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA.