Quantitative high-throughput screening data analysis: challenges and recent advances

Abstract

In vitro HTS holds much potential to advance drug discovery and provide cell-based alternatives for toxicity testing. In quantitative HTS, concentration–response data can be generated simultaneously for thousands of different compounds and mixtures. However, nonlinear modeling in these multiple-concentration assays presents important statistical challenges that are not problematic for linear models. Importantly, parameter estimation with the widely used Hill equation model is highly variable when using standard designs. Failure to consider parameter estimate uncertainty properly would greatly hinder chemical genomics and toxicity testing efforts. In this light, optimal study designs should be developed to improve nonlinear parameter estimation; or alternative approaches with reliable performance characteristics should be used to describe concentration–response profiles.

Introduction

HTS of compounds is the primary force driving the transformation of drug discovery [1] and toxicity testing [2,3]. Robotic plate handling enables traditional HTS to screen thousands of chemicals at a single compound concentration to identify a small number of candidates with a desired biological activity [4]. Quantitative HTS (qHTS) assays, a recent development in HTS technology, perform multiple-concentration experiments in a low-volume cellular system (e.g. <10 μl per well in 1536-well plates) using high-sensitivity detectors [5]. In qHTS, large chemical libraries are screened for lead candidates with the prospect of lower false-positive and false-negative rates than traditional HTS approaches [5]. However, discordant results from a recent analysis [6] of two high-profile pharmacological studies [7,8] have raised questions about the effectiveness of qHTS to establish reliable predictors of drug response or to identify the mechanism of a drug. Similar skepticism has been directed toward ToxCast™ Phase I bioactivity data after failed attempts to predict in vivo hazard from high-throughput in vitro screening data [9].

qHTS datasets can include thousands of response profiles. For example, the US Tox21 collaboration can simultaneously test more than 10 000 chemicals across 15 concentrations [10]. Pre-specified statistical models are usually fitted to the data to estimate parameters that, in turn, can be used to rank chemicals by activity level or serve as input data for prediction modeling [9] or association mapping [11]. Accordingly, the disappointing results observed in some large-scale in vitro screening experiments can be caused, in part, by the limitations of nonlinear least squares parameter estimation within standard qHTS study designs. The Hill equation [12] (hereafter HEQN) is by far the most common nonlinear model used to describe qHTS response profiles and serves to illustrate the challenges of parameter estimation for this data type.

The Hill equation in qHTS

The HEQN has a long and well-deserved reputation for accurately describing concentration– response or exposure–response relationships in biochemistry, pharmacology and hazard prediction. Although the HEQN is not a realistic reaction scheme for a receptor with more than one binding site [13], it usually fits sigmoidal concentration–response data reasonably well. HEQN parameters have convenient biological interpretations and direct comparison of its parameter estimates from different experiments provides a convenient way to compare concentration–response profiles. The logistic form [14] of the HEQN is given in Equation 1.

$$R_i=E_0+\frac{\left(E_{\infty }-E_0\right)}{1+exp\{-h[log{C}_i-log{\text{AC}}_{50}]\}}$$ [Equation 1]

The measured response R_i at concentration C_i is expressed as a function of the baseline response E₀, the maximal response E_∞, the concentration for half-maximal response AC₅₀ and the shape parameter h. The logarithm in Equation 1 ensures that back-calculated estimates of AC₅₀ from logAC₅₀ are restricted to positive values. The values of AC₅₀ and E_max (E_∞– E₀) calculated from Equation 1 are frequently used in pharmacological research and toxicological assessments as approximations for compound potency and efficacy, respectively, but other parameters can also provide useful information [11,15,16]. AC₅₀ is often used to prioritize chemicals for further studies and commonly serves as the basis for prediction modeling. Although E_max can be important when allosteric effects are a potential concern in candidate selection, the estimation of AC₅₀ from the HEQN will receive special attention below.

In spite of the advantages mentioned above, curve fits to the HEQN are not equally reliable across the diverse range of response profiles that characterize high-throughput in vitro screening. Not every substance has a sigmoidal concentration–response curve within the tested concentration range. For instance, ‘flat’ in vitro assay profiles with full responses relative to a positive control represent the most potent chemicals in the entire screen; yet these compounds will often generate poor fits to the HEQN and be declared inactive (false negatives). Alternatively, concentration–response profiles of truly null compounds could, simply as a result of random variation in response, be fitted well by Equation 1 within the tested concentration range and then spuriously be declared active (false positives). To make matters worse, non-monotonic relationships in the data could express real biology [17], which cannot be adequately described by the inherently monotone HEQN. Consequently, it is important to make activity calls using approaches known to have reliable classification performance across a broad range of possible profiles [18–20].

Parameter estimation

Parameter estimates obtained from the HEQN can be highly variable if the range of tested concentrations fails to include at least one of the two HEQN asymptotes, responses are heteroscedastic or concentration spacing is suboptimal [11,16,18–20]. Consider Figure 1, which evaluates the repeatability of AC₅₀ estimates in simulated 14-point concentration–response curves for cases with only one asymptote (when AC₅₀ = 0.001 μM or 10 μM) or cases with two asymptotes (when AC₅₀ = 0.1 μM). Concentration–response data for 2000 monotonically increasing curves were simulated from Equation 1 using an error variance set to ‘5% of the positive control response’ at each concentration with three different values of E_max [19]. In all cases, E₀ (the baseline response) was set to ‘0’ and h (the Hill slope) was set to ‘1’. In this setting, the effects of curve shape are explored through different AC₅₀ values and signal-to-noise ratio is investigated with different E_max values. Figure 1 shows the mean and 95% confidence intervals (2.5th to 97.5th percentiles) describing the distribution of AC₅₀ estimates derived from model fits to Equation 1 for all 2000 simulated profiles. Estimates of AC₅₀ were precise (i.e. had narrow confidence intervals) when the concentration range defined both asymptotes (AC₅₀ = 0.1 μM and E_max ≥50%) or when the lower asymptote was established (AC₅₀ = 10 μM with E_max = 100%). _[s1]In the other instances the estimates of AC₅₀ showed very poor repeatability, even spanning several orders of magnitudes in some cases.

Figure 1.

_[s3]Mean and 95% confidence intervals (2.5th to 97.5th percentiles) of the distribution of estimated AC₅₀ obtained from 2000 concentration–response datasets simulated from the logistic Hill equation with 5% error at each response level. True values of E_max (% positive control response) and AC₅₀ (in μM) are shown in the figure. Vertical dotted lines indicate true AC₅₀ in each simulation study. All simulated curves assume E₀ = 0% and h = 1. Representative curves for cases with one asymptote (AC₅₀ = 0.001 μM or 10 μM) and two asymptotes (AC₅₀ = 0.1 μM) are shown on the left side of the figure. The solid curves correspond to the true model from which the data were generated and the dots indicate simulated responses. The HEQN model was fitted to the data using nonlinear least squares.

Random measurement error impacts observed response levels, which can seriously diminish the reproducibility of parameter estimates [11,18–20]. Including experimental replicates can improve measurement precision [4]. Table 1 illustrates the impact of increased sample size on parameter estimation for the simulation study described above. Overall, larger sample sizes lead to noticeable increases in the precision of AC₅₀ and E_max estimates. Unfortunately, systematic error can be introduced into the data at numerous levels. The well location of the chemicals within plates can vary from run to run, compound purity can degrade over time, signal can bleach across wells (signal flare) and compounds can be inadvertently transferred between plates (compound carryover) [4,10]. The possibilities of biased response measurements challenge the notion that separate screening runs are true experimental replicates. How to incorporate data from multiple runs into a single substance-specific model remains an open research question. In practice, however, nonlinear model fits of qHTS data are often restricted to single replicate chemical profiles even when data from multiple experiments are available.

Table 1. The effect of sample size on parameter estimation in simulated datasets^a.

True AC50b	True Emaxc	nd	μe and [95% CIf] for AC50 estimates	μe and [95% CIf] for Emax estimates
0.001	25	1	7.92e−05 [4.26e−13, 1.47e+04]	1.51e+03 [−2.85e+03, 3.1e+03]
		3	4.70e−05 [9.12e−11, 2.42e+01]	30.23 [−94.07, 154.52]
		5	7.24e−05 [1.13e−09, 4.63]	26.08 [−16.82, 68.98]
0.001	50	1	6.18e−05 [4.69e−10, 8.14]	50.21 [45.77, 54.74]
		3	1.74e−04 [5.59e−08, 0.54]	50.03 [44.90, 55.17]
		5	2.91e−04 [5.84e−07, 0.15]	50.05 [47.54, 52.57]
0.001	100	1	1.99e−04 [7.05e−08, 0.56]	85.92 [−1.16e+03, 1.33e+03]
		3	5.52e−04 [1.00e−05, 0.03]	100.01 [95.45, 104.57]
		5	7.24e−04 [4.94e−05, 0.01]	100.04 [95.53, 104.56]
0.1	25	1	0.09 [1.82e−05, 418.28]	97.14 [−157.31, 223.48]
		3	0.10 [0.03, 0.39]	25.53 [5.71, 45.25]
		5	0.10 [0.05, 0.20]	24.78 [−4.71, 54.26]
0.1	50	1	0.10 [0.04, 0.23]	50.64 [12.29, 88.99]
		3	0.10 [0.06, 0.16]	50.07 [46.44, 53.71]
		5	0.10 [0.04, 0.23]	50.05 [47.72, 52.36]
0.1	100	1	0.10 [0.07, 0.15]	99.97 [93.57, 106.36]
		3	0.10 [0.06, 0.16]	100.06 [97.32, 102.79]
		5	0.10 [0.08, 0.13]	100.00 [94.96, 105.03]
10	25	1	247.66 [9.03e−05, 6.79e+08]	1.72e+04 [−3.05e+04, 3.70e+04]
		3	39.42 [3.26e−03, 4.77e+05]	7.59e+03 [−1.02e+04, 1.17e+04]
		5	18.47 [0.06, 5.31e+03]	417.56 [−7.06e+03, 7.89e+03]
10	50	1	23.84 [0.04, 1.46e+04]	334.47 [−3.31e+04, 3.98e+03]
		3	11.72 [2.12, 64.69]	59.86 [−2.50e+03, 3.70e+03]
		5	10.54 [5.35, 20.74]	51.20 [37.97, 64.43]
10	100	1	10.99 [3.12, 38.65]	104.57 [50.88, 158.25]
		3	10.17 [2.88, 35.89]	101.14 [84.99, 117.29]
		5	10.05 [5.60, 18.06]	100.48 [88.57, 112.39]

^aThe distribution of estimated AC₅₀ (in μM) and E_max (% positive control) was obtained by generating 2000 concentration-response datasets from the logistic Hill equation, given by Equation 1 in the text, with 5% error at each response level for three different sample sizes n.

^bThe true value of AC₅₀ in the simulated dataset.

^cThe true value of E_max in the simulated dataset.

^dValues of n refer to the number of replicates at each tested concentration.

^eThe mean value of an estimated parameter.

^fThe 95% confidence interval (CI) for each parameter.

Although parameter estimates can lie well beyond the experimental concentration range (Figure 1), constraining model fits so that parameter estimates lie within the tested concentration range can be misleading because the true values are not known. Parameter constraints guarantee that there is no extrapolation beyond the range of the data, but they cannot remove large associated errors in parameter estimates and will produce systematic distortion if the constraints are false. Confidence intervals for nonlinear parameter estimates are typically based on parameter estimate standard errors. These errors are approximated by a procedure that utilizes the first order derivative in the Taylor series expansion of the nonlinear model. If the second order term (or curvature effect) is important, the standard errors could be unreliable [16,21,22]. Furthermore, the activity call process could bias confidence intervals for selected parameters. Correcting selected parameter confidence intervals is referred to as adjusting for selection and multiplicity [23]. In view of these considerations, caution should be used in generating and interpreting parameter confidence intervals.

Parameter estimation can also be sensitive to outliers and influential observations. As a result, many analysis routines attempt to identify and remove outliers strategically before curve fitting. Statistical methods for outlier detection generally require plenty of replicate points [24], which is problematic given the small sample sizes (typically n ≤3 replicates per concentration level) characterizing most qHTS studies. Some outlier removal routines in qHTS are based on procedures that transform a non-monotonic profile into a sigmoidal curve. However, efforts to define outliers by assuming a specific model form a priori could bias estimates because the true model governing the concentration–response curve is not known in advance. Lim et al. used an estimation procedure, termed M-estimation, that is robust to potential outliers [18]. Unlike ordinary least squares, which minimizes the sum of squared residuals, the proposed M-estimators decrease the importance of particular residual components during curve fitting by minimizing a weighted function of the residuals.

Statistical significance in nonlinear regression

Applications of qHTS often focus on selecting potentially active compounds and ranking hits for downstream analysis. Manual inspection of the data can be prohibitively time consuming, subjective and susceptible to human error [19]. It is good practice to visualize curve fits on multiple-curve displays to understand the computational output. However, such graphics could be misleading when confidence intervals of parameter estimates are not accurate, resulting in very high false-positive rates [16]. Although tested chemicals should not be discarded from all future inquiry based on the statistical assessment of a primary screen, proper statistical testing can provide a reliable framework for selecting candidates.

Unlike linear modeling, nonlinear regression estimators do not have analytical solutions and statistical inference is not based on exact distributional theory [21]. Approaches based on resampling methods provide one possible route for obtaining statistical significance. Unfortunately, nonlinear least squares requires a large number of starting values to achieve appropriate model fits, often involving a substantial computational time for each tested chemical [25]. Lim et al. argue that at least 100 000 bootstraps or permutations would be needed per chemical to obtain good estimates of small P-values [18]. Therefore, conventional resampling methods appear impractical for qHTS studies containing several thousand chemicals. However, it might be possible to develop adaptive bootstrapping procedures that can effectively estimate small P-values with considerably fewer bootstraps than conventional approaches [26].

In lieu of resampling methods, asymptotic theory (or large sample theory) can be used to calculate P-values [21]. Asymptotic theory states that, as sample sizes increase, the gap between the true distribution of a statistic and the approximate distribution of a statistic will asymptote to zero. However, when multiple test correction is used to correct for the condition that many statistical tests are being conducted simultaneously (one for each chemical in the assay), it is only the very small P-values that will lead to statistical significance. Because asymptotic approximations do a poor job of estimating very small P-values, standard procedures might not be able to control false discovery rate. Still, a recent simulation study based on preliminary test estimation methodology provides reasonable false discovery rate control and good power [18].

Concluding remarks and future outlook

Important matters remain to be settled in the field of qHTS data analysis. Data standardization [6], novel assay construction [6], robust methods for statistical testing [18] and feature selection in classification modeling [27,28] should continue to be researched in order to establish standards of best practice. However, I do not believe that the problematic qHTS results can be resolved as long as we rely on parameter estimates derived from naive curve-fitting strategies. The problem does not lie in the estimation procedures themselves but rather in the application of nonlinear modeling to study designs lacking suitable concentration spacing and sufficient replication to describe the full concentration-response curves [18,21,29,30].

Optimal designs (ξ) can be used to specify the concentrations best suited to estimate parameters for an assumed model structure to achieve maximal precision and reduced experimental costs compared with nonoptimal designs [21,29,30]. The support points for ξ consist of the k concentrations (C₁, C₂, …, C_k) and weights (w₁, w₂, …, w_k) that can account for heteroscedasticity in the data. D-optimal designs deliver the most precise parameter estimates by minimizing a measure of the asymptotic variance-covariance matrix of estimated parameter values Σ (i.e. minimizing the determinant of Σ). Designs based on the D-optimality criterion have been applied to the HEQN model to improve the precision of parameter estimation compared with uniformly spaced concentrations [29,30]. The parameter variance–covariance matrix corresponding to Equation 1 is given below:

$$\sum =\left[\begin{array}{cccc}\text{Var}\left(E_0\right)& \text{Cov}(E_0,E_{\infty })& \text{Cov}(E_0,log{\text{AC}}_{50})& \text{Cov}(E_0,h)\\ \text{Cov}(E_{\infty },E_0)& \text{Var}\left(E_{\infty }\right)& \text{Cov}(E_{\infty },log{\text{AC}}_{50})& \text{Cov}(E_{\infty },h)\\ \text{Cov}(log{\text{AC}}_{50},E_0)& \text{Cov}(log{\text{AC}}_{50},E_{\infty })& \text{Var}(log{\text{AC}}_{50})& \text{Cov}(log{\text{AC}}_{50},h)\\ \text{Cov}(h,E_0)& \text{Cov}(h,E_{\infty })& \text{Cov}(h,log{\text{AC}}_{50})& \text{Var}\left(h\right)\end{array}\right]$$ [Eqn 2]

Where $\sqrt{\text{Var}\left(E_0\right)}$, $\sqrt{\text{Var}\left(E_{\infty }\right)}$, $\sqrt{\text{Var}(log{\text{AC}}_{50})}$ and $\sqrt{\text{Var}\left(h\right)}$ represent the approximate standard errors in parameters E₀, E_∞, logAC₅₀ and h, respectively.

As a general rule, Σ is derived from the inverse of the Fisher information matrix M, where Σ is proportional to M^-1 = (V^TΩV)^-1. Each element of the extended design matrix V is a k × p matrix consisting of a first-order partial derivative of the assumed model with respect to parameter p evaluated at concentration k, and Ω is a diagonal matrix of the support point weights [21]. However, when the calculation of M^-1 is sensitive to small perturbations in the data (i.e. when the condition number of M is large) the inversion of M is not reliable and singularities can arise in which the determinant of Σ cannot be computed. Problems with matrix inversion can greatly reduce the accuracy and precision of parameter estimation and yield inaccurate P-values [16]. Ridge-regression approaches that penalize the size of regression coefficients provide a potential solution to this difficulty [18]. Also, the dependence of Σ on assumed model parameters has important implications for qHTS studies, where each tested chemical is expected to have a unique set of parameters. D-optimal designs for the Hill model were previously found to be robust to incorrectly assumed parameter values [29]. If this result does not hold up for qHTS, it might be possible to find optimal designs by using a Bayesian approach with a suitable prior distribution of ξ characterizing a wide range of chemicals [16,18].

Many parameter estimates in qHTS will inevitably be associated with large standard errors and new methods should be developed that consider parameter estimate uncertainty in the analysis. Alternatively, qHTS analysis should be based on metrics that have more-reliable performance characteristics than parameters estimated from nonlinear model fits. For instance, the weighted entropy score (WES), representing the average activity level of a concentration–response curve, has been shown to outperform AC₅₀ in ranking chemical profiles by activity level [20]. Another potentially robust metric is the area under the curve (AUC), a measure of compound activity computed as the area between the concentration–response curve and the X-axis [11]. Although drug-response data from two recent pharmacogenomics studies [7,8] produced largely inconsistent results [6], AUC was more concordant between studies than IC₅₀ estimates based on comparisons of Spearman's rank correlation [6]. No overall statistical difference between the AUC and IC₅₀ metrics was detected in that report [6], but experimental disparities between the original studies [7,8] might have hindered the utility of this comparison. In any event, recognizing the limitations of standard nonlinear modeling in qHTS data analysis is necessary to reach a workable solution to the challenges posed by inconsistent results in pharmacogenomics and in vitro toxicity studies.

Highlights.

• Quantitative HTS has much potential to advance drug discovery and toxicity testing

• Nonlinear modeling of concentration-response data in suboptimal designs is unreliable

• The Hill equation is widely used to model quantitative HTS data

• Large uncertainties in parameter estimation often accompany nonlinear model fits to the data

• Robust methods should be developed to take full advantage of quantitative HTS technology