A two-stage experimental design for dilution assays

Abstract

Dilution assays to determine solute concentration have found wide use in biomedical research. Many dilution assays return imprecise concentration estimates because they are only done to orders of magnitude. Previous statistical work has focused on how to design efficient experiments that can return more precise estimates, however this work has not considered the practical difficulties of implementing these designs in the laboratory. We developed a two-stage experiment with a first stage that obtains an order of magnitude estimate and a second stage that concentrates effort on the most informative dilution to increase estimator precision. We show using simulations and an empirical example that the best two-stage experimental designs yield estimates that are remarkably more accurate than standard methods with equivalent effort. This work demonstrates how to utilize previous advances in experimental design in a manner consistent with current laboratory practice. We expect that multi-stage designs will prove to be useful for obtaining precise estimates with minimal experimental effort.

1 |. INTRODUCTION

The serial limiting dilution assay (SLDA) is a widely used method for determining the concentration of particles in a volume by repeatedly diluting a specimen until no particles remain. Dilution assays are used when an assay is only able to detect the presence or absence of a target such the concentration of infectious virus in a sample (Lee et al., 2015), or the density of immunocompetent cells in a specimen (Greenwood and Yule, 1917). The design of these experiments requires determining the number of dilution levels and replicates at each dilution. Previous work on the experimental design of SLDAs has focused on exploiting a priori information about the concentration of target virus to determine these parameters (Strijbosch et al., 1988; Matthews, 1998; MacKen, 1999; Atkinson et al., 2007). However, this information is often not available to experimentalists, and it is not clear how robust these estimates are when the design assumptions are unreliable.

There exists a well-developed literature on the design of SLDAs. The program DESIGN (Strijbosch et al., 1988) was developed to aid experimenters in applying this theory in their assays. This program requires determining an interval bound for the concentrations and a range of positive responses (defined as an assay where virus is present) that are deemed by the experimenter to be sufficiently informative, with statistical theory providing some guidance as to what fraction of positive responses can be considered highly informative (De St. Groth, 1982). The program then outputs an efficient experimental design. Using this software is a valuable exercise for scientists interested in the quantitative machinery underlying the study of experimental design, however, determining these inputs can be inconvenient for the typical experimentalist. As a result, those tasked with carrying out SLDAs have underutilized this and similar work focused on the design of efficient experiments.

We developed an adaptive, two-stage design aimed to be easily implementable and to provide precise estimates of concentration. Our approach builds on commonly used experimental methods and on the theory of efficient SLDA design. However, in contrast to previous work, our method does not require more quantitative skills than a typical experiment. Instead, we focus on extending commonly used approaches to provide improved estimates. We show that our two-stage method yields precise estimates with relatively little increase in effort and we provide code for performing two-stage experimental design and estimation in the R software environment (R Core Team, 2015).

2 |. MODELS & METHODS

2.1 |. Estimation

The general goal of an SDLA experiment is to determine the concentration of a target molecule from a presence-absence assay by successively diluting a specimen. In our particular case, the goal will be to estimate the viral titer, defined as the concentration of infectious virus in a specimen. We denote the unknown concentration as λ. Under the single-hit Poisson model (Taswell, 1981), the unknown number of viral infectious units is modeled as a Poisson random variable with rate parameter, λd_i. This rate is the expected number of infectious units in the diluted sample when the original sample is diluted by the factor of 1/d_i. The dilution, d_i, is the proportion of the original sample present in the diluted sample. This model assumes that any diluted samples with one or more infectious units have a positive response. The probability that the specimen contains no infectious units is given by $p=e^{-\lambda d_i}$. The number of replicates at dilution d_i that contain one or more infectious units is modeled as a binomial distribution with probability $1-e^{-\lambda d_i}$.

There are several existing ways to estimate λ from these data. Commonly used estimation techniques include the Reed-Muench method (Reed and Muench, 1938), Spearman-Karber method (Spearman, 1908), maximum-likelihood (ML) estimation (Fisher, 1922), and alternative forms of minimum chisquare estimation (Taswell, 1981). Here we used ML estimation due to it’s appealing statistical properties (e.g., consistent estimators and asymptotically normal sampling distributions) and because the single-hit Poisson model can be expressed as a generalized linear model (GLM) using the complementary log-log link function (Finney, 1951). This link function has the added advantage that the sampling distribution of the transformed estimate, ln $(\widehat{\lambda })$, is well approximated as a normal distribution, thus making Wald-type confidence intervals estimates more robust (De St. Groth, 1982). We denote the estimate of λ as $\widehat{\lambda }$.

Because GLMs are a commonly used statistical technique, many statistical software packages can be used to obtain these estimates. Code to estimate λ in the R software environment from an SLDA is provided in Box 1 and in the supporting information. Other widely used software such as the web-interface ELDA (Hu and Smyth, 2009), the popular L-Calc software (Stem Cell Technologies, Vancouver, BC, Canada), the R package SLDAssay (Trumble et al., 2017), which also allows for bias corrected MLE’s, can also be used to make these estimates.

2.2 |. Experimental design

The quality of an estimate from an SLDA is contingent on the design of the experiment. Previous work looking at the efficient designs of SLDAs require an a priori estimate of the concentration, λ, to focus experimental effort on an informative range of dilutions. The Fisher information, which describes the amount of information that a data point will provide for an unknown parameter (Bain and Engelhardt, 2000), of the single-hit Poisson model is given by $\mathcal{I}(p)=np\ln {(p)}^2/(1-p)$. This quantity can be maximized numerically, showing that the most efficient experiment is one corresponding to the dilution with a positive response to the virus 80% of the time (p ≈ 0.2). This optimal response corresponds to the highest variance in the estimator, $\widehat{\lambda }$. In contrast, standard dose-response curves have maximal information at a response rate of 50%.

De St. Groth (1982) proposed to use the Fisher information to design experiments around “highly informative” dilutions that correspond to an expected viral response between 60% and 90%. Strijbosch et al. (1987) proposed a similar solution for SLDAs when an a priori range of concentrations can be determined. This method requires determining lower and upper limits of the fraction of nonresponsive wells considered to be “sufficiently informative”, though it is left to the user to determine what is informative. These fractions are then used to define a range of dilutions. Finally, Zelterman et al. (2010) proposed using maximum entropy as an alternative to the Fisher information; this accounts for the expected variation in the fraction of nonresponsive wells at each level of dilution. These approaches require experimenters to determine a range of plausible values for λ.

Unfortunately, the information needed to apply these designs will often not be available before conducting an experiment and can yield dilutions that are inconvenient to implement in practice. To overcome these issues, we propose a two-stage approach, where the first stage is used to derive an order of magnitude estimate for λ and the second stage is used to refine the first stage estimate.

The first stage of our procedure follows the standard SLDA approach, starting with 10-fold dilutions over several orders of magnitude done in replicate following the guidelines from Goldman and Green (2015). We then apply the GLM described in the above Estimation section to estimate the concentration, ${\widehat{\lambda }}_1$. We use ${\widehat{\lambda }}_1$ to design a second round of more precise experiments by using the dilution that is most informative for λ based on the first estimate ${\widehat{\lambda }}_1$. This informative dilution corresponds to a well-response rate of 80%. From the Fisher information this dilution is given by ${\widehat{d}}_{\text{opt}}=-\ln (1-0.8)/{\widehat{\lambda }}_1\approx 1.61/{\widehat{\lambda }}_1$. We call this two-step process the “optimal Fisher” design.

The optimal Fisher design can yield dilutions that are inconvenient to implement in practice. Therefore, we also propose an “approximate Fisher” design that rounds ${\widehat{d}}_{\text{opt}}$ to a dilution that is easy to implement. For example, if ${\widehat{d}}_{\text{opt}}=1.54\cdot {10}^{-5}$ and 10-fold dilutions are straightforward to implement then we would use ${\widehat{d}}_{\text{app}}={10}^{-5}$. Neither the optimal nor approximate Fisher designs take into account the uncertainty in the first stage estimate, ${\widehat{\lambda }}_1$.

In order to determine whether we need to incorporate the uncertainty present in the first-stage estimate, ${\widehat{\lambda }}_1$, into the design of the second-stage we consider additional designs that focus effort in a range around ${\widehat{d}}_{\text{opt}}$. We follow De St. Groth (1982) by considering a range of dilutions that fall within 90% of the maximum value of the Fisher information. This informative range is defined rather arbitrarily but is important to note that it gives a dilution range that depends only on ${\widehat{\lambda }}_1$, independent of the sample size used in the first stage. De St. Groth (1982) showed that this region of the Fisher information corresponds to a design that is expected to yield positive responses between 60% (at dilution $d_{\min }=-\ln (1-0.6)/{\widehat{\lambda }}_1$) and 90% (at dilution $d_{\max }=-\ln (1-0.9)/{\widehat{\lambda }}_1$). We designed the experiment assuming the dilutions are equally spaced on the log of this interval. We call this experimental design the “informative range” design.

We also consider a range determined by the 95% confidence interval of the estimator, ${\widehat{\lambda }}_1$. In this case, the design depends on the precision of the first stage of the experiment. This corresponds to a range of $d_{\min }=-\ln (0.2)/{\widehat{\lambda }}_1\left({\alpha }_{0.975}\right)$ to $d_{\max }=-\ln (0.2)/{\widehat{\lambda }}_1\left({\alpha }_{0,025}\right)$, where ${\widehat{\lambda }}_1\left({\alpha }_{0.975}\right)$ and ${\widehat{\lambda }}_1\left({\alpha }_{0,025}\right)$ are the estimates of the upper and lower 95% confidence intervals of $\widehat{\lambda }$ estimated using the profile likelihood. We call this the “confidence interval” design.

A two-stage design can be thought of as a sequential process, with the second round of experiments conditional on the first. Previous authors (McCullagh, 1981; Ford et al., 1989) have pointed out that sequential procedures do not affect the likelihood function, which may be written as if the observations are independent. Thus, the final estimate for λ combines the first-and second-stage datasets and using standard estimation approaches.

2.3 |. Assay experiment

To illustrate the two-stage approach we estimated the titer of human rhinovirus 1B (RV1B) in cell culture using an SLDA. A standard endpoint dilution assay was performed using RV1B (ATCC VR-1645) in HeLa cells (ATCC CRL-1958) (Lee et al., 2015). We conducted the first stage of SLDA using a 0.1 ml sample of RV1B then diluting this sample by a factor of 0.5. We followed this initial dilution by serial 10-fold dilutions (Table 1), repeating this process in triplicate for a total of 24 samples. The viral dilutions were added to HeLa cells in a 96-well plate and incubated at 37 °C. Cell death (i.e., cytopathic effects) was monitored daily, and final endpoints were determined after five days. We applied the single-hit Poisson model to this first stage to determine estimates to use in the second stage Fisher optimal and Fisher approximate designs. In the second stage of the Fisher optimal design, RV1B was diluted to the to 8.75 · 10⁻⁶ and in the Fisher approximate design it was diluted to 5 · 10⁻⁶, We added these dilutions to 24 replicate wells of HeLa cells. Finally, we repeated the first stage experiment in order to obtain an estimate that had the same number of samples as the two-stage designs. We call this design the standard design. Cytopathic effects were monitored for five days and stained with crystal violet to visualize the final endpoints (Figure S1). Code to implement the optimal Fisher and approximate Fisher designs and estimate λ in this RV1B example are provided in Box 1 and in the supporting information.

TABLE 1.

Factors fixed and varied in the simulation studies.

Factor varied	Factors held fixed
Number of replicates (sample size, n) varied from 2 to 8 (n = 32 to 128)	True sample concentration (λ = 106) Relative effort (equal number of replicates used in stage 1 and stage 2)
Relative effort (Number of replicates in stage 2 varied from 1 to 6)	True sample concentration (λ = 106), Total sample size (n = 64)
True sample concentration (λ varied from 103 to 107)	Number of replicates and relative effort (16 samples in each stage)

2.4 |. Simulation study

We used three sets of simulations to compare the performance of all the strategies considered in the Experimental design section. Our first simulation experiment tested the effect of the number of replications on estimate accuracy and precision by varying the number of replications at each level of dilution from 2 to 8. This corresponded to between 14 and 56 samples for each stage. The first stage experiment covered a wide range of 10-fold dilutions from 10⁻¹ to 10⁻⁸. We generated data at each dilution from the single-hit Poisson model with λ = 10⁶. Simulations were repeated 10⁵ times at each level of replication. Estimates of the concentration from the first stage experiments, ${\widehat{\lambda }}_1$, determined the strategy of the second stage using the rules detailed in the Experimental design section. The second stage designs all used the same total sample size as the first stage. However, the number of dilutions in the second stage depended on the particular design as discussed in the Experimental design section. We also implemented a design using the same dilutions as the first stage, but with double the sample size at each dilution. We refer to this as the standard design. The standard design allowed us to control for the effects of sample size on estimates.

Our second simulation study explored how much effort should be placed into the second stage of the experiment relative to the first stage. In these simulations we fixed the total number of samples at 64 and number of dilutions at 8. We then varied the number of samples in the first and second stages of the experiment by systematically varying the number of replicates (n) in the first stage from 2 to 7. We then placed the remaining effort into the second stage of the experiment so that the total proportion of samples in the second stage are given by (64–8· n)/64, where n is the number of replicates.

Our third and final simulation study fixed the number of replicates at 2 and varied the true value of the infectious viral concentration, λ, from a titer of 10³ to 10⁷ over five equally spaced values on the natural log-scale of this interval in order to determine whether the relative performance of each design depends on the titer. Simulation and estimation were performed as in the previous experiment. We repeated simulations 10⁴ times at each value of λ.

For each of the simulation experiments described above, we estimated the concentration, λ, in R using the GLM function (Code in Box 1 and Appendix 1). We determined estimator accuracy using the bias of the estimate $\widehat{\lambda }$, given as $\text{Bias}=\widehat{\lambda }-\lambda$, where λ was the true value used in simulations and the estimated parameter is given by $\widehat{\lambda }$. Estimator precision was determined using the standard error, defined as $\text{SE}=\sqrt{{\sum }_{i=1}^n(\widehat{\lambda }-E(\widehat{\lambda }){)}^2/n}$. The root-mean-square error (RMSE) combines both accuracy and precision into a single measure and is given by $\text{RMSE}=\sqrt{\text{Bias}+{\text{SE}}^2}$. This quantity is commonly used to measure the quality of a biased estimator (Lehmann and Casella, 1998).

3 |. RESULTS

Example of a two-stage SLDA

Here we step through the design and analysis of a two-stage SLDA experiment. Details on the experimental procedure are given in the Methods.

• Conduct the first stage of the SLDA. The range of dilutions should be wide enough that one or more viral infectious units will be in the first dilution but none will be in the final dilution (e.g., stage 1 in Figure 1). Here we used three replicates at each dilution.

• Estimate the viral infectious unit concentration, λ₁, from the first stage data. The estimation of the experimental results given in Table S1 can be implemented in R as:

  > glm0 = glm(Response ~ offset(log(Dilution)),
   family=binomial(link="cloglog"),
    weights=Replicates, data=Table1)
  > lambda = exp(glm0$coef)
  > print(lambda)
  (Intercept)
     183835.1
  > print(exp(confint(glm0)))
    2.5% 97.5%
  39308 611208
  

• Conduct the second stage of the SLDA. According to the optimal Fisher design we will place all effort at d_opt = −ln(0.2)/1.84 · 10⁵ = 8.75 · 10⁻⁶. For the approximate Fisher design we used d_app = 5 · 10⁻⁶.

• Now estimate λ from the full dataset with stages 1 and 2 using the same GLM code as above. For the optimal Fisher design this gives ${\widehat{\lambda }}_{\text{opt}}=2.02\cdot {10}^5$ and a 95% confidence interval of (1.24·10⁵, 3.18·10⁵). Piont estimates and 95% confidence intervals of the concentration for the Stage 1, standard, optimal Fisher, approximate Fisher designs are given in Figure 2. To control for the effects of sample size we compared these estimates to a standard design that used the same number of dilutions as the Stage 1 design but doubled the number of replicates. Data and code to replicate these analyses are provided in the supporting information.

FIGURE 1.

Example of the design and results of a two-stage dilution assay experiment. Each circle corresponds to an assay well. Numbers above each circle in stage 1 correspond to the dilution used for each of the 3 replicates. Red circles are a positive response, indicating the presence of RV1B titer. Blue circles are clear of the virus. In stage 1 the assay is conducted across a wide range of dilutions, in stage 2 all replicated are conducted at the dilution corresponding to a positive response of 80% (here estimated to be 8.75 · 10⁻⁶). This figure appears in color in the electronic version of this article, and color refers to that version.

FIGURE 2.

Point estimates and 95% confidence intervals from the SLDA experiment using four different experimental designs. Stage 1 used 10-fold dilutions over a range of eight dilutions and three replications per dilution for 24 total samples. The standard design uses the same eight dilutions with six replications per dilution (48 samples). The second stage of the optimal- and approximate-Fisher designs used 24 samples. This figure appears in color in the electronic version of this article.

3.1 |. Simulation study results

Our comparison of the four different experimental designs showed that two-stage designs performed better than the standard design under all experimental conditions. In the first simulation experiment the optimal Fisher design performed best in terms of the RMSE over all sample sizes (Figure 3C). In our simulations the relative performance of estimates from the standard design and from the two-stage designs was approximately constant (Figure 3C). For the conditions in this simulations study, using the optimal Fisher design over the standard design reduces the RMSE by at least half at any sample size. For comparison, the optimal Fisher design with two replicates, corresponding to 32 total samples, was more precise (lower bias, Figure 3A) and accurate (lower standard error, Figure 3B) than a standard design with 80 total samples. The overall measure of estimator quality, the RMSE, was dominated by the standard error. In addition, the bias behaved slightly differently than the standard error and RMSE (Figure 3A). The approximate Fisher design performed noticeably worse overall than the optimal Fisher design, though the bias of the approximate Fisher design was the lowest out of all criterion at low and high sample sizes. The two-stage designs that accounted for uncertainty in parameter estimates performed nearly as well as the optimal Fisher design and better than the approximate Fisher design. The informative range design was slightly more precise and accurate than the confidence interval design (Figure 3A and 3B), because the informative range design uses a narrower range of dilutions than the confidence interval design for the sample sizes explored in these simulations. As the sample size increased, differences between the all two-stage estimates decreased.

FIGURE 3.

The effect of replication number on estimator accuracy measured by the estimator bias (panel A), precision measured by the standard error (panel B), and the overall measure of estimator quality the root-mean-square error (panel C) in the estimated virus concentration $(\widehat{\lambda })$ as the number of replications per dilution is increased form 2 (total sample size of 32) to 8 (total sample size of 128). This figure appears in color in the electronic version of this article.

When varying the amount of effort placed into the stage 1 and stage 2 portions of the experiment we found that estimator precision was optimized when slightly more effort was placed into stage 2 relative to stage 1 (Figure 4). Bias was minimized in both of the Fisher designs when 75% (the maximum explored in our simulations) of the total effort was placed into the second stage (Figure 4A). The standard error and mean-square error were minimized for both designs when 62.5% of effort was placed into the second stage (Figure 4B and 4C).

FIGURE 4.

The effect of changing the relative amount of effort placed into the first and second stages of the experiment on estimator accuracy measured by the estimator bias (panel A), precision measured by the standard error (panel B), and overall estimator quality measured by the root-mean-square error (panel C) in the estimated virus concentration $(\widehat{\lambda })$. The proportion of effort out of 64 total samples with eight dilutions is 1 − n/8 where n is the number of replicates used in stage 1. This figure appears in color in the electronic version of this article.

When varying the concentration, λ, our results were consistent with the previous set of simulations. For all concentrations, estimates from the optimal Fisher design performed best for both estimator precision and accuracy (Figure 5A and 5B). The two-stage designs that accounted for estimator uncertainty did perform slightly better than the approximate Fisher design. However all two-stage designs performed much better than the standard design (Figure 5C).

FIGURE 5.

The effect of the true concentration, λ, on estimator accuracy measured by the estimator bias (panel A), precision measured by the standard error(panel B), and the root-mean-square error (panel C) in the estimated virus concentration $(\widehat{\lambda })$. All simulations used 32 samples. This figure appears in color in the electronic version of this article.

4 |. DISCUSSION

We developed an approach to conducting SLDAs that builds on both standard lab practice and on past work designing efficient assays. Our two-stage method was designed with practitioners in mind and yields more precise estimates with less effort and materials than traditional methods. For example, we found through simulations that the mean-square error of the optimal Fisher design with 32 samples was lower than estimates using the standard method with 80 samples. We were also able to confirm these results in practice. In our empirical example we estimated standard errors that were 50% smaller than a standard design with the same sample size, a result consistent with predictions from our simulation study. In contrast to our approach, previous work has focused on SDLA designs that are efficient in the statistical sense (e.g., De St. Groth, 1982; Strijbosch et al., 1987; Matthews, 1998; Zelterman et al., 2010), but the requirements to implement these designs may often be impractical for everyday use by scientists.

Despite the compelling results of our study, there are some scenarios a two-stage approach may not be appropriate. Time-consuming experimental techniques such as the viral outgrowth assays used to determine residual HIV infections (Rosenbloom et al., 2015) may be better served by placing more effort up front and performing lots of dilutions, rather than a two-stage approach. In these cases, the increase in time to perform a two-stage assay may outweigh the benefits of improved estimator precision. Other situations where two-stage designs may be unnecessary are when the concentration of the target is well-known a priori. For example, longitudinal studies have repeatedly shown that antibody concentration can persist long after the clearance of an infection (Heininger et al., 2004; Le et al., 2004; Gulbudak et al., 2017). Thus, information about when the acute phase of the infection occurred can be used to design precise assays. In such cases, a two-stage approach would not be necessary as the first stage dilutions can be focused enough to yield precise estimates. However, a singular feature of biological systems is the astounding amount of variation that occurs even in highly controlled environments. For example, in an experiment exposing mice to a respiratory virus, Gonzalez et al. (2018) found significant individual heterogeneity in titer as well as variability depending on the time since infection. Thus, we expect that there will be many cases where two-stage designs provide a quick and reliable improvement over standard assay designs.

Previous work exploring the tradeoff between the number of dilutions and the number of replicates to use in an experiment found that having more dilutions and fewer replications generally led to better estimators by reducing estimator bias and RMSE (MacKen, 1999). We expanded on this work by looking at how effort should be partitioned in a two-stage experimental design and found that experimental effort is better allocated to the second stage of the experiment (Figure 4). However, we note that the reduction of the RMSE of this unequal effort distribution was quite small when compared to the reduction made to carrying out a two-stage design in the first place (Figure 3).

Our design assumes that the mechanism generating data is the single-hit Poisson model, which assumes that the variation in positive responses at each dilution as arising due purely to sampling error. However, errors that occur when measuring the dilution may invalidate the single-hit Poisson model, leading to increases in model complexity (Higgins et al., 1998; Gelman et al., 2004) that are not commonly used in practice. It is likely that significant measurement error would lead to changes in the Fisher information, thus specifying the optimal Fisher design would depend on knowing the amount of measurement error present. However, we expect that our two-stage design will be robust to moderate levels of measurement error as replication is expected to alleviate the effects of nonsystematic measurement error.

Work on the statistical design of SLDAs has long focused on applying a priori information to constrain sampling effort to a range of highly informative dilutions. Our approach builds off past efforts through a two-stage process that allows experimenters to focus their effort as more information becomes available through the experiment. We believe that multi-stage designs will provide a useful bridge between the theoretical literature on efficient assays for labs that need to obtain precise estimates of virus concentrations with minimal materials and effort.