Methods for comparing multiple digital PCR experiments

Abstract

The estimated mean copy per partition (λ) is the essential information from a digital PCR (dPCR) experiment because λ can be used to calculate the target concentration in a sample. However, little information is available how to statistically compare dPCR runs of multiple runs or reduplicates. The comparison of λ values from several runs is a multiple comparison problem, which can be solved using the binary structure of dPCR data. We propose and evaluate two novel methods based on Generalized Linear Models (GLM) and Multiple Ratio Tests (MRT) for comparison of digital PCR experiments. We enriched our MRT framework with computation of simultaneous confidence intervals suitable for comparing multiple dPCR runs. The evaluation of both statistical methods support that MRT is faster and more robust for dPCR experiments performed in large scale. Our theoretical results were confirmed by the analysis of dPCR measurements of dilution series.

Both methods were implemented in the dpcR package (v. 0.2) for the open source R statistical computing environment.

Introduction

Digital PCR (dPCR) is a PCR-based method, which enables a precise quantification of nucleic acids. The conventional PCR performs single reaction per one sample. However, in the case of dPCR the sample is separated into a large number of partitions, in which the reaction is carried out individually (clonal amplification). The partitioning allows to assess the number of amplified template molecules by detection of their presence (positive call) or absence (negative call) in particular partitions [1], [2]. Since the output of these results is binary, we do not know if the positive partition contains one or more template molecules. The Poisson transformation is required to compute the average number of template molecules per partition, expressed by λ:

$$\lambda =-log\left(1-\frac{k}{n}\right)$$ (1)

where k is number of positive partitions and n is number of negative partitions.

Thanks to that, it is possible to measure precisely concentrations of nucleic acids with high sensitivity and reliability. Therefore, dPCR found common applications in amplification of DNA samples for next-generation sequencing and detection of variation in genomic sequences, e.g. point mutations and repeats [1].

In contrast to the conventional PCR, in which the number of amplification cycles ideally is proportional to the initial copy number, dPCR does not depend on the cycle number to determine the initial amount of nucleic acids in the sample. In particular, the quantitative real-time PCR is known to be demanding regarding preprocessing, quantification cycle determination and multi-plate measurements [3], [4], [5], [6]. The dPCR methodology eliminates the dependence on the exponential shape of data to estimate the concentration of target nucleic acids and enables their absolute quantification. Therefore, this method does not need calibration curves and may even be less susceptible to inhibitors. The amplification chemistry of absolute quantification in the dPCR is orchestrated by well established methods such as analogue PCR or isothermal amplification [7], [2], [8], [9], [10].

Precision, sensitivity, dynamic range, number of partitions and their volume are important parameters in a dPCR system [11]. Moreover, technical replicates are affected by different intrinsic and extrinsic influences increasing the variation of obtained results. This variation needs to be assessed to make a valid statement about the assay performance. As all diagnostic methods, the dPCR requires tools to check consistency of obtained results. There is a growing need for statistical methods for the analysis and design of experiments using digital PCR experiments.

Previously, two methods to compute the λ value and its uncertainty were described. Dube's approach uses confidence intervals [12], whereas Bhat's method is based on the uncertainty [13]. The latter is not a confidence interval in the statistical sense, but nevertheless can be employed to compute probability coverage of the estimated λ value. The Dube's method computes binomial confidence intervals for proportion k/n using the method of normal approximation. Briefly, the binomial distribution of positive counts with the parameters p = k/n and n trials is approximated by a normal distribution. Both Bhat's and Dube's methodologies do not address multiple comparisons of runs, which is a common task during the design and analysis of dPCR experiments.

Here, we propose two approaches for the comparison of multiple dPCR experiments. Both are able to simultaneously compare the λ values of multiple runs. One of them is based on Generalized Linear Models and the second one is the uniformly most powerful ratio test combined with multiple testing correction. Our findings were implemented in the R statistical computing environment [14], which has numerous functionalities devoted to analysis of dPCR and qPCR reactions [15].

Methods

Generalized Linear Models – GLM

Generalized Linear Models (GLM) are linear models for data in which the response variable may have a non-normal distribution (e.g. binomial distribution of positive partitions in the case of dPCR experiments). We employ a simplistic model reflecting the relationships between variables in dPCR results, given by formula:

$$logY={\beta }^{T}X$$ (2)

where Y are counts of positive partitions, X are experiments names (categorical data) and β are coefficients for every run. Since the binomially distributed response is explained by the linear combination of parameters (in our specific case experiment names) we call such model binomial regression as described in detail elsewhere [16]. Briefly, we employed the logarithm function (function that limits values of response) and the estimated means of copies per partitions by calculating $\hat{\lambda }=exp\beta$. Importantly, our GLM employs the quasibinomial model, which describes the binomial distribution with excessive zeros. That means that number of zeros may surpass its value predicted by the binomial distribution [17].

The GLM model used in this analysis and represented by Eq. (2) can be refined by adding further effects, such as the technical replication. This may decrease the variance within replicate experiments.

Next the differences between estimated coefficients are evaluated pairwise [18], [19]. The resulting p-values require no posterior correction because the familywise error is controlled. This approach is a single-step procedure, because the decision (rejection or acceptance of null hypothesis) is not based on the decision of another hypothesis. As implied by the name, all tests are made in the single step, independently and simultaneously.

Multiple testing

The $\hat{\lambda }$ from two or more dPCR experiments may be pairwise compared using the uniformly most powerful (UMP) ratio test. Uniformly most powerful tests have the highest statistical power (probability that the test correctly rejects the false null hypothesis H₀) for all tests with the same α. The p-values are computed using the TST (twice the smaller tail) method [20], [21]. Similarly to the GLM method, this is a single-step approach. However, to control the familywise error rate the pairwise comparison requires an appropriate adjustment of p-values, as the Benjamini–Hochberg correction [22]. The UMP ratio test has the following null-hypothesis:

$$H_0:\frac{{\lambda }_1}{{\lambda }_2}=1$$ (3)

where λ₁ and λ₂ are mean numbers of template molecules per partition in two experiments. It is also possible to employ other tests (e.g., proportion test) designed to determine the probabilities of having positive partitions [23].

Instead of relying on confidence intervals (CI) computed by the UMP test, we used CIs calculated by the Wilson method. It was proven [24] that Wilson's CIs have generally a good coverage probability. In particular, this applies when the probability of success is close to zero, such as in cases of dPCR experiments where the concentration of the template is low. The Wilson's confidence intervals are calculated independently for every dPCR run and its significance level is adjusted using the Dunn–Šidák correction [25]:

$${\alpha }_{\mathrm{adj}}=1-{(1-\alpha )}^{1/T}$$ (4)

where α is the significance level and T is the number of tested hypotheses.

Such intervals are wider than non-corrected CIs but they simultaneously contain the mean number of template molecules per partition of several experiments at a given significance level. Due to this, simultaneous confidence intervals allow testing the results against thresholds while keeping the familywise error at specified level. In other words, the value of λ is statistically different (on the significance level equal to the confidence level) from the threshold value only if the threshold value is not covered by the confidence intervals.

For clarity, we decided to call our novel approach Multiple Ratio Test (MRT).

Confidence intervals

The confidence intervals computed by MRT were compared to uncertainty intervals calculated using two other methods: Bhat's [13] and Dube's [12]. The formulation of considered methods is presented in Table 1.

Table 1.

Formulations of uncertainty intervals of compared methods: MRT, Dube and Bhat. Φ: cumulative normal distribution function.

Method name	Formulation
MRT	$-log\left(\frac{1}{1+\frac{1}{n}{\left({\mathrm{\Phi }}^{-1}\left(1-\frac{\alpha }{2}\right)\right)}^2}\left(\frac{k}{n}+\frac{{\left({\mathrm{\Phi }}^{-1}\left(1-\frac{\alpha }{2}\right)\right)}^2}{2n}\pm {\mathrm{\Phi }}^{-1}\left(1-\frac{\alpha }{2}\right)\sqrt{\frac{k^2}{n}\left(1-\frac{k}{n}\right)+\frac{{\left({\mathrm{\Phi }}^{-1}\left(1-\frac{\alpha }{2}\right)\right)}^2}{4n^2}}\right)\right)$
Bhat	$\lambda \pm \sqrt{\frac{\frac{k}{n}}{n(1-\frac{k}{n})}}$
Dube	$-log\left(1-\frac{k}{n}\pm {\mathrm{\Phi }}^{-1}(1-\frac{\alpha }{2})\sqrt{\frac{\frac{k}{n}(1-\frac{k}{n})}{n}}\right)$

Evaluation of statistical methods

Two implemented approaches, GLM and MRT were compared over 2 millions in silico dPCR runs. Each simulation contained two groups with the same number of runs (from one to four runs), which started from the same number of template molecules. The simulation was repeated 60 times (Fig. 1). Both frameworks were used to compare the runs. The GLM approach used a quasibinomial model.

Figure 1.

Scheme of the in silico dPCR for two approaches based on Generalized Linear Models (GLM) and Multiple Ratio Test (MRT). The groups (A and B) contain one to four runs (coloured circles). Each run within one group is generated using the same number of molecules in 1000 partitions. The number of molecules (m₀) range from 10 to 200. Equally-sized groups were finally compared by GLM and MRT methods. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

Experiment setup

To validate the proposed MRT analysis framework, we performed a dPCR experiment on dilution series using the dPCRmethyl dataset from the dpcR package [26]. This dataset contains a dilution of methylated human gDNA (see package for details). For each concentration level, three samples were measured.

Implementation

All findings were implemented in the dpcR package (v. 0.2) [26] for the R statistical computing environment. The GLM uses functionality from multcomp package [27] and the ratio for MRT comes from rateratio.test [28]

The stable version of the package is available from https://cran.r-project.org/web/packages/dpcR/index.html. Plots were created with ggplot2 [29]. All analyzes were done in dedicated R environments as described previously [15]. An executable R script is attached as a supplement.

Results and discussion

Monte Carlo evaluation

The statistical tests showed that neither GLM nor MRT rejected the true H₀ (Eq. (3)) in all cases. Both methods had ‘blind spots’, in which they could not distinguish runs containing different numbers of template molecules (Fig. 2). The number of non-rejected hypotheses widened with the growth of the λ values. For example, the difference between a run with λ = 0.01 and runs with λ only up to 0.03 was not considered statistically significant by at least one method, whereas a run with λ = 0.14 differed significantly from runs with λ even up to 0.20.

Figure 2.

Monte Carlo evaluation of the GLM and MRT methods for number of runs in a single group from one to four. The colour of the tile describes which of these methods distinguished between two compared groups (at the level α = 0.05). Abscissa and ordinate represent the average number of template molecules per partition used in the in silico experiment in equinumerous groups A and B, respectively. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

The statistical power of the GLM and MRT methods depended also on the number of runs (Figs. 2 and S1, and Table S1). The GLM was more sensitive than MRT to changes of the template concentration when a single run per group was compared. In turn, the MRT had higher statistical power when tested scenarios involving more than one run per group. This is in line with the definition of a multiple comparison.

The speed of both approaches was assessed in a separate benchmark. The MRT method was always faster than GLM regardless of the number of runs compared. The difference considerably increased non-linearly with the number of runs, whereas the number of partitions had much smaller impact on the computation time (Fig. 3 and Table S2).

Figure 3.

The impact of number of runs on the average time of data analysis by GLM and MRT methods. The MRT outperforms GLM for larger data sets. The performance does not depend on the total number of partitions per run, n. The average time was calculated from 100 repetitions.

Coverage probability of confidence intervals

We performed 1×10⁶ dPCR experiments in silico (2×10⁴ droplets each) for the range of λ values from 0.05 to 2.00 with the increment 0.05 (1.2×10⁷ experiments total).

To assess the simultaneous coverage probability, we randomly divided experiments into 2000 groups (500 experiments each) for all values of λ. We counted the frequency of groups in which all confidence intervals contained the true value of λ.

The adjusted confidence intervals, which are much wider than nominal/uncorrected CIs, guaranteed 0.95 stable simultaneous coverage probability. This offers more reliable comparison of multiple runs (Table S3).

Analysis of experimental data

We analyzed experimental data using the MRT. The proposed framework was able to distinguish between different concentrations even on restrictive confidence levels. As in the Monte Carlo evaluation, there are ‘blind spots’, where MRT is not able to discriminate between two relative concentrations, for example between 0% and 5% (Table 2, detailed results in Fig. S2 and Table S4).

Table 2.

Fraction of significant pairwise comparisons of dPCR results.

MRT yields sufficiently high p-values when comparing the same concentrations with the only exception for the 75% concentration. In this case, only one sample was wrongly assessed as having the λ value different from other samples with the same concentration.

The confidence intervals determined by MRT always covered the real mean number of template molecules per partition. This indicates that all technical replicates come from the same source (Fig. 4 and Table S5). On the opposite, the real λ was not covered in 22% and 44% of cases by the confidence intervals computed using Dube's and Bhat's method, respectively.

Figure 4.

The confidence intervals of the results of the experiment. The red color marks confidence intervals that failed to cover the real value of λ denoted by a green line. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

Conclusions

The MRT framework seems to be a tailor-made method for comparing multiple dPCR experiments. This method is characterized by a statistical power which is superior to the simplistic GLM approach and is considerably faster. Performance is very important, because even the analysis of large experimental setups should be possible on average desktop computers.

Adjusted confidence intervals suitable for multiple comparison problems are an integral part of MRT. Although Dube's method might be also enhanced by Dunn–Šidák correction for the purpose of multiple testing, it uses normal approximation to compute the confidence intervals. Others have shown that this approach is inferior to Wilson's method as employed by MRT, because its probability coverage is inadequate for values of k/n close to 0 and 1 [24]. In consequence, for low and large values of λ, the real probability coverage of confidence intervals determined by Dube's method may differ from its nominal value in contrast to the Wilson's method.

It is worth to note that both analyzed statistical methods may be further enhanced. GLM may be expanded with more complex models taking more effects as recently shown into account [9]. MRT can utilize different tests for binomial proportion (e.g., test of equal proportions).

Both tests are available by the test_counts function of the dpcR package the commonly available open source statistical software environment R (see Fig. 5).

Figure 5.

Example for the analysis of multiple dPCR runs. In this example, the graphical user interface RKWard [30] (v. 0.6.5) was used to compare multiple dPCR runs. The principle analysis of such experiments has been described in [15]. (A) Only a few lines of code are needed to load, process and report the results of the dPCR experiments. Here we used the dPCRmethyl dataset contained in the dpcR package. (B) The results of the analysis can easily be presented as interactive table. (C) Plots of the analysis can be created as needed. In our example, we used the functionality of the dpcR package to create an plot automatically.

Appendix A. Supplementary data

The following are the supplementary data to this article:

The code used to perform analysis of the experimental data.

Monte Carlo evaluation of the GLM and MRT methods for number of runs in a single group from one to four. The colour of the tile describes the range to which p-value belongs (0–0.001, 0.001–0.05, 0.05–1). Two first ranges indicate significant results. The number inside each cell is the exact p-value.

The confidence intervals of the results of the experiment. Each chart consists of p-values (represented by dots) of MRT performed on runs from two different relative concentrations. For example, top left chart contains p-values of tests comparing runs with 100% and 0% of relative template concentration. The red line denotes confidence level 0.05.

Monte Carlo evaluation of the GLM and MRT methods for number of runs in a single group from one to four. Columns describe respectively: number of runs in the single group in the comparison, the λ value in group 1, the λ value in group 2, average p-value computed using GLM, average p-value computed using MRT.

The impact of number of runs on the average time of data analysis by GLM and MRT methods. Columns describe: number of runs in the single group in the comparison, number of partitions in the single run, mean time of data analysis by GLM (in seconds), mean time of data analysis by MRT (in seconds).

Probability of λ coverage based on confidence intervals calculated by adjusted Dunn–Šidák correction in comparison to Bhat's and Dube's methods. Columns describe respectively: λ value, type of the probability coverage, confidence intervals provided by MRT method, confidence intervals provided by Bhat's method, confidence intervals provided by Dube's method.

The table contains the p-values of the experiment results. Columns describe: dilution of sample 1, dilution of sample 2, p-value computed by MRT.

The confidence intervals of the experiment results. Columns describe: dilution of sample, method of computing CIs, λ value, lower border of CI, upper border of CI, assumed λ value.